Tracking down cause of reconciliation loop continuously firing?

[empath-nirvana]

My reconcile loop fires continuously, and I can't figure out why. Is there anyway to figure out which object exactly triggered a loop and why?

[empath-nirvana]

Second question: Should patching the status trigger another loop if the status hasn't actually changed?

[nightkr]

No, Kubernetes should generally not trigger an update for no-ops.

[clux]

No, but be careful about writing non-deterministic fields to status though, like last updated timestamps. These will cause a new resource version and a new hit to the reconciler (even if the source of the change was the reconciler itself).

[nightkr]

Controller logs the reason for the reconciliation being triggered. For example:

2022-08-02T15:14:39.303347Z  INFO reconciling object{object.ref=ZookeeperCluster.v1alpha1.zookeeper.stackable.tech/simple-zk.default object.reason=object updated}: stackable_zookeeper_operator::zk_controller: Starting reconcile

Here, object.reason=object updated means that the reconciliation was triggered by the object itself being changed.

[empath-nirvana]

This is why I don’t understand — the object I have in the logs hasn’t had a change, but it’s in the logs triggering updates thousands of times. The resource version is changing though — I suspect because of status updates.

On Wed, Aug 3, 2022 at 4:47 PM Teo Klestrup Röijezon < ***@***.***> wrote: Controller logs the reason for the reconciliation being triggered. For example: 2022-08-02T15:14:39.303347Z INFO reconciling object{object.ref=ZookeeperCluster.v1alpha1.zookeeper.stackable.tech/simple-zk.default object.reason=object updated}: stackable_zookeeper_operator::zk_controller: Starting reconcile Here, object.reason=object updated means that the reconciliation was triggered by the object itself being changed. — Reply to this email directly, view it on GitHub <#975 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUVYTLZACFB3ILIWOER4XUTVXLLEXANCNFSM55QF3D7A> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- “The world is not comprehensible, but it is embraceable.”

[nightkr]

Could there be some oscillation going on? One useful trick can be to run kubectl get [type]/[obj] --watch --output=yaml | tee foo and diff the revisions pairwise.

[empath-nirvana]

There's no difference -- i'm watching the object spin by and literally the only thing changing is the resourceVersion. And I did diff successive versions to verify.

[empath-nirvana]

Is there a way a to filter out status updates, because the thing triggering the change is what is supposed to be an idempotent status patch.

[clux]

Sometimes you just get repeat events (when say the connection is lost and you need to reset the state), and it's impossible to avoid all of them.

Currently smart logic for this requires some user action to cache. There is PR for something similar (to look at say observedGeneration) in #911, but it does not hook into the Controller.

[empath-nirvana]

It's just just random extra events, it's cycling through hundreds of times a minute.