Operator sends k8 controller garbagecollector into an infinite loop

[andrewwebber]

I have a 'coreos-baremetal' cluster that when a etcd-cluster is launched by the etcd-operator to k8-controller goes crazy.

This eventually causes the k8-controller node to run out of disk space.
Workaround currently is to hourly kill and restart the controller container :(

{"log":"I0118 09:51:08.806215       1 event.go:217] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"10.10.2.102\", UID:\"882b86ac-dc0d-11e6-9ff0-001e4f520ea4\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RegisteredNode' Node 10.10.2.102 event: Registered Node 10.10.2.102 in NodeController\n","stream":"stderr","time":"2017-01-18T09:51:08.8087232Z"}
{"log":"I0118 09:51:08.806251       1 event.go:217] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"10.10.3.103\", UID:\"db9e7d24-dc0d-11e6-9ff0-001e4f520ea4\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RegisteredNode' Node 10.10.3.103 event: Registered Node 10.10.3.103 in NodeController\n","stream":"stderr","time":"2017-01-18T09:51:08.808748906Z"}
{"log":"I0118 09:51:08.806268       1 event.go:217] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"10.10.3.104\", UID:\"d856a908-dc0d-11e6-9ff0-001e4f520ea4\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RegisteredNode' Node 10.10.3.104 event: Registered Node 10.10.3.104 in NodeController\n","stream":"stderr","time":"2017-01-18T09:51:08.808759381Z"}
{"log":"I0118 09:51:08.806282       1 event.go:217] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"10.10.3.202\", UID:\"d175e1b3-dc0d-11e6-9ff0-001e4f520ea4\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RegisteredNode' Node 10.10.3.202 event: Registered Node 10.10.3.202 in NodeController\n","stream":"stderr","time":"2017-01-18T09:51:08.808769116Z"}
{"log":"I0118 09:51:08.806296       1 event.go:217] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"10.10.3.204\", UID:\"c8bca4db-dc0d-11e6-9ff0-001e4f520ea4\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RegisteredNode' Node 10.10.3.204 event: Registered Node 10.10.3.204 in NodeController\n","stream":"stderr","time":"2017-01-18T09:51:08.808778906Z"}
{"log":"I0118 09:51:08.806310       1 event.go:217] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"10.10.3.205\", UID:\"d54de405-dc0d-11e6-9ff0-001e4f520ea4\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RegisteredNode' Node 10.10.3.205 event: Registered Node 10.10.3.205 in NodeController\n","stream":"stderr","time":"2017-01-18T09:51:08.808788798Z"}
{"log":"I0118 09:51:08.806324       1 event.go:217] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"10.10.3.102\", UID:\"d18c2b99-dc0d-11e6-9ff0-001e4f520ea4\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RegisteredNode' Node 10.10.3.102 event: Registered Node 10.10.3.102 in NodeController\n","stream":"stderr","time":"2017-01-18T09:51:08.808798638Z"}
{"log":"I0118 09:51:08.806338       1 event.go:217] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"10.10.3.101\", UID:\"db492f64-dc0d-11e6-9ff0-001e4f520ea4\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RegisteredNode' Node 10.10.3.101 event: Registered Node 10.10.3.101 in NodeController\n","stream":"stderr","time":"2017-01-18T09:51:08.808808629Z"}
{"log":"I0118 09:51:08.806352       1 event.go:217] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"10.10.3.201\", UID:\"d50dc664-dc0d-11e6-9ff0-001e4f520ea4\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RegisteredNode' Node 10.10.3.201 event: Registered Node 10.10.3.201 in NodeController\n","stream":"stderr","time":"2017-01-18T09:51:08.80881979Z"}
{"log":"I0118 09:51:08.806368       1 event.go:217] Event(api.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"10.10.3.203\", UID:\"0eea82fc-dc0e-11e6-9ff0-001e4f520ea4\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RegisteredNode' Node 10.10.3.203 event: Registered Node 10.10.3.203 in NodeController\n","stream":"stderr","time":"2017-01-18T09:51:08.808830097Z"}
{"log":"I0118 09:51:18.471149       1 garbagecollector.go:780] Garbage Collector: All monitored resources synced. Proceeding to collect garbage\n","stream":"stderr","time":"2017-01-18T09:51:18.471400353Z"}
{"log":"E0118 09:58:30.196257       1 garbagecollector.go:593] Error syncing item \u0026garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:metatypes.OwnerReference{APIVersion:\"coreos.com/v1\", Kind:\"EtcdCluster\", UID:\"a9df64c0-dd64-11e6-a5cc-001e4f520ea4\", Name:\"etcd-client\", Controller:(*bool)(0xc421948230)}, Namespace:\"rhino-ci\"}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, dependents:map[*garbagecollector.node]struct {}{(*garbagecollector.node)(0xc422b50510):struct {}{}}, owners:[]metatypes.OwnerReference(nil)}: unable to get REST mapping for kind: EtcdCluster, version: coreos.com/v1\n","stream":"stderr","time":"2017-01-18T09:58:30.196391795Z"}
{"log":"E0118 09:58:30.197389       1 garbagecollector.go:593] Error syncing item \u0026garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:metatypes.OwnerReference{APIVersion:\"coreos.com/v1\", Kind:\"EtcdCluster\", UID:\"a9df64c0-dd64-11e6-a5cc-001e4f520ea4\", Name:\"etcd-client\", Controller:(*bool)(0xc421948230)}, Namespace:\"rhino-ci\"}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, dependents:map[*garbagecollector.node]struct {}{(*garbagecollector.node)(0xc422b50510):struct {}{}}, owners:[]metatypes.OwnerReference(nil)}: unable to get REST mapping for kind: EtcdCluster, version: coreos.com/v1\n","stream":"stderr","time":"2017-01-18T09:58:30.197454872Z"}
{"log":"E0118 09:58:30.197714       1 garbagecollector.go:593] Error syncing item \u0026garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:metatypes.OwnerReference{APIVersion:\"coreos.com/v1\", Kind:\"EtcdCluster\", UID:\"a9df64c0-dd64-11e6-a5cc-001e4f520ea4\", Name:\"etcd-client\", Controller:(*bool)(0xc421948230)}, Namespace:\"rhino-ci\"}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, dependents:map[*garbagecollector.node]struct {}{(*garbagecollector.node)(0xc422b50510):struct {}{}}, owners:[]metatypes.OwnerReference(nil)}: unable to get REST mapping for kind: EtcdCluster, version: coreos.com/v1\n","stream":"stderr","time":"2017-01-18T09:58:30.197757981Z"}
{"log":"E0118 09:58:30.198055       1 garbagecollector.go:593] Error syncing item \u0026garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:metatypes.OwnerReference{APIVersion:\"coreos.com/v1\", Kind:\"EtcdCluster\", UID:\"a9df64c0-dd64-11e6-a5cc-001e4f520ea4\", Name:\"etcd-client\", Controller:(*bool)(0xc421948230)}, Namespace:\"rhino-ci\"}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, dependents:map[*garbagecollector.node]struct {}{(*garbagecollector.node)(0xc422b50510):struct {}{}}, owners:[]metatypes.OwnerReference(nil)}: unable to get REST mapping for kind: EtcdCluster, version: coreos.com/v1\n","stream":"stderr","time":"2017-01-18T09:58:30.198092124Z"}
{"log":"E0118 09:58:30.198427       1 garbagecollector.go:593] Error syncing item \u0026garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:metatypes.OwnerReference{APIVersion:\"coreos.com/v1\", Kind:\"EtcdCluster\", UID:\"a9df64c0-dd64-11e6-a5cc-001e4f520ea4\", Name:\"etcd-client\", Controller:(*bool)(0xc421948230)}, Namespace:\"rhino-ci\"}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, dependents:map[*garbagecollector.node]struct {}{(*garbagecollector.node)(0xc422b50510):struct {}{}}, owners:[]metatypes.OwnerReference(nil)}: unable to get REST mapping for kind: EtcdCluster, version: coreos.com/v1\n","stream":"stderr","time":"2017-01-18T09:58:30.198735295Z"}
{"log":"E0118 09:58:30.199169       1 garbagecollector.go:593] Error syncing item \u0026garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:metatypes.OwnerReference{APIVersion:\"coreos.com/v1\", Kind:\"EtcdCluster\", UID:\"a9df64c0-dd64-11e6-a5cc-001e4f520ea4\", Name:\"etcd-client\", Controller:(*bool)(0xc421948230)}, Namespace:\"rhino-ci\"}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, dependents:map[*garbagecollector.node]struct {}{(*garbagecollector.node)(0xc422b50510):struct {}{}}, owners:[]metatypes.OwnerReference(nil)}: unable to get REST mapping for kind: EtcdCluster, version: coreos.com/v1\n","stream":"stderr","time":"2017-01-18T09:58:30.19922236Z"}

[hongchaodeng]

This is known issue in k8s 1.5 regression: #501 (comment), kubernetes/kubernetes#39816

We don't support k8s 1.5 until upstream fixes it.

[narayanan]

Is there a known work around to manage the situation? Currently this is eventually bringing down the k8s cluster.

[hongchaodeng]

@narayanan

See the discussion at kubernetes/kubernetes#39816 .
This reduces to one line log in k8s 1.6 . It's planned to fix dynamic mapping in 1.7 .

[narayanan]

@hongchaodeng Thanks will try out operator with 1.6 beta