Operator is trying to get pods on cluster scope #743
Description
Hi Everyone,
We have observed the following behavior on one of clusters where the operator is failing to update ArangoDeployment during deployment inspection
When it fails, it constantly trying to list the pods on the cluster scope. and become sort of "stuck in a loop" trying to get the pods from the cluster scope. which results in the operator become non-responsive.
upon restart of the operator pod, this no longer occurs.
Environment
kubernetes 1.19.7 via AKS.
kube-arangodb operator version is 1.15
Logs
| Date | Event |
|---|---|
| 6/15/2021, 10:08:43.121 AM | 2021-06-15T10:08:43Z DBG Inspect deployment... component=deployment deployment=arangodb-cluster operator-id=ssk6j |
| 6/15/2021, 10:08:43.214 AM | 2021-06-15T10:08:43Z WRN Member is not ready for long time, but it is not safe to mark it a failed because: TODO component=deployment deployment=arangodb-cluster id=SNGL-ben2cwi1 operator-id=ssk6j role=single |
| 6/15/2021, 10:08:43.220 AM | 2021-06-15T10:08:43Z INF Ensuring labels component=deployment deployment=arangodb-cluster operator-id=ssk6j |
| 6/15/2021, 10:08:43.220 AM | 2021-06-15T10:08:43Z INF Ensuring annotations |
| 6/15/2021, 10:08:43.266 AM | 2021-06-15T10:08:43Z INF Reconciliation loop took 76.365279ms component=deployment deployment=arangodb-cluster operator-id=ssk6j |
| 6/15/2021, 10:08:43.266 AM | 2021-06-15T10:08:43Z DBG ...inspected deployment component=deployment deployment=arangodb-cluster interval=10s operator-id=ssk6j |
| 6/15/2021, 10:08:43.266 AM | 2021-06-15T10:08:43Z INF Inspect loop took 144.685529ms component=deployment deployment=arangodb-cluster operator-id=ssk6j |
| 6/15/2021, 10:08:53.000 AM | 2021-06-15T10:08:53Z DBG Inspect deployment... component=deployment deployment=arangodb-cluster operator-id=ssk6j |
| 6/15/2021, 10:08:53.542 AM | 2021-06-15T10:08:53Z INF Event(v1.ObjectReference{Kind:"ArangoDeployment", Namespace:"", Name:"", UID:"", APIVersion:"database.arangodb.com/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'Reconcilation Failed' Unable to update UpToDate condition: Unable to update condition: failed to patch ArangoDeployment status: resource name may not be empty operator-id=ssk6j |
| 6/15/2021, 10:08:53.542 AM | 2021-06-15T10:08:53Z INF Updated condition component=deployment condition=UpToDate deployment=arangodb-cluster operator-id=ssk6j reason="Spec Changed" status=false |
| 6/15/2021, 10:08:53.542 AM | 2021-06-15T10:08:53Z DBG failed to patch ArangoDeployment status error="resource name may not be empty" component=deployment deployment=arangodb-cluster operator-id=ssk6j |
| 6/15/2021, 10:08:53.542 AM | 2021-06-15T10:08:53Z DBG ...inspected deployment component=deployment deployment=arangodb-cluster interval=250ms operator-id=ssk6j |
| 6/15/2021, 10:08:53.542 AM | E0615 10:08:53.541997 1 event.go:265] Unable to write event: 'can't create an event with namespace 'default' in namespace 'kube-arangodb'' (may retry after sleeping) |
| 6/15/2021, 10:08:53.542 AM | 2021-06-15T10:08:53Z INF Reconciliation loop took 261.701µs component=deployment deployment=arangodb-cluster operator-id=ssk6j |
| 6/15/2021, 10:08:53.542 AM | 2021-06-15T10:08:53Z INF Inspect loop took 275.079405ms component=deployment deployment=arangodb-cluster operator-id=ssk6j |
| 6/15/2021, 10:08:53.792 AM | 2021-06-15T10:08:53Z DBG Inspect deployment... component=deployment deployment=arangodb-cluster operator-id=ssk6j |
| 6/15/2021, 10:08:53.795 AM | 2021-06-15T10:08:53Z INF Inspect loop took 2.78901ms component=deployment deployment=arangodb-cluster operator-id=ssk6j |
| 6/15/2021, 10:08:53.795 AM | 2021-06-15T10:08:53Z ERR Unable to get resources error="pods is forbidden: User "system:serviceaccount:kube-arangodb:arango-arangodb-cluster-operator" cannot list resource "pods" in API group "" at the cluster scope" component=deployment deployment=arangodb-cluster operator-id=ssk6j |
| 6/15/2021, 10:08:53.795 AM | 2021-06-15T10:08:53Z DBG ...inspected deployment component=deployment deployment=arangodb-cluster interval=250ms operator-id=ssk6j |
| 6/15/2021, 10:08:54.045 AM | 2021-06-15T10:08:54Z DBG Inspect deployment... component=deployment deployment=arangodb-cluster operator-id=ssk6j |