controller should debounce reconcile requests

[ctron]

It looks to me as if the controller triggers exactly one event for each change that was detected on e.g. an "owned resource".

Assume you have 10 owned resources for each resource, and 10 managed resources. That makes 100 events and thus 100 reconciliations on the startup of an operator.

The same for scheduled/delayed reconcile outcomes.

IMHO the controller queue should queue up a reconcile request if there is no event yet scheduled for this resource, or if the scheduled event is further in the future than the new request.

EDIT: (clux): See #318 (comment)

[clux]

Revisiting this old issue to ensure we are doing everything we can. Here is a brief overview of what we do:

• deduplication in scheduler. yes, provided it happened before it ran
• debouncing of events by a few seconds in the scheduler. could be better - moved to Set a controller debounce time #1247
• allow opt-in predicates to hard-filter per-input stream

For anything else there are still the standard approaches;

• use annotations on secondary streams to set checkpoints for the reconciler
• use status fields on the primary stream to create checkpoints for the reconciler
• gate check in the reconciler work that is guaranteed to be done to avoid duplicate heavy work

but these approaches are scarier since you run the risk of creating less idempotent reconcilers that can get into bad states (reconciler thinks it is done, and you cannot re-trigger it because it gate checks itself out of correcting the problem).

Debouncing

I feel the key question left for this issue is:
Is our debouncing as good as it can be? Judging by the code comments, probably not, but this stuff is also hard.
Ideally we want classical debouncing behavior:

• event comes in? schedule a reconciliation for the relevant object at now() + N seconds
• another event comes in before reconciliation is due? reschedule, for now() + N seconds
• N second elapses after last event? reconcile

5 seconds is probably OK given the heavy work that usually works, but if we ever manage to expose options for the Controller, then this would be a good thing to expose.

Leaving this open and resolving older comments that are now out of date.
If someone wants to look at refining this, or experiment, you are welcome to, otherwise going to leave this as is.

[clux]

Closing this. Better scoped issue about how to improve scheduler debouncing linked in #1247