[addon-operator] fix: make module hook registration idempotent and race-free#797
Merged
Merged
Conversation
b592a4c to
c22479a
Compare
Glitchy-Sheep
added a commit
to deckhouse/deckhouse
that referenced
this pull request
Jul 10, 2026
…ifetime fixes Picks up shell-operator 38c7e4f (Error log + binding_monitor_missing_total in SnapshotsFor) and ed5d828 (shared informer factory lifetime owned by the manager + factory_informer_dead_total), and addon-operator 74703bc (Synchronization retry test, fail-on-unqueued-sync-task fix, shell-operator pin bump). Root and tools modules together, so 'go generate' stays green. Pseudo-versions to be replaced with release tags once the upstream PRs merge (flant/shell-operator#922, flant/addon-operator#797). Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- AddHook replaces the previously registered hook object in all bindings instead of appending a duplicate: a stale entry keeps a nil HookController and crashes the kube-events dispatcher after a registration retry - registered/controllersReady flags are read and written under the storage lock, giving a happens-before edge between setting hook controllers and observing the readiness flag - RegisterHooks is serialized with a dedicated mutex so two concurrent ModuleRun tasks cannot register the same module twice - getHooks sorts and returns a copy instead of the shared slice Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
One hook whose monitor could not start aborted HandleModuleEnableKubernetesBindings for the whole module, so the remaining hooks never emitted their Synchronization contexts. Collect errors with errors.Join instead of returning on the first one. Ported from #798. Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
The best-effort enable loop (errors.Join) is only safe together with the idempotent EnableKubernetesBindings from shell-operator c52b11d: on an older shell-operator, pass 1 starts monitors for every successful hook and the retry then hits the alreadyEnabled early-return, which yields zero Synchronization contexts for all of them - a wider loss window than before the aggregation. Re-point to the tagged release once flant/shell-operator#922 merges. Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Encodes the acceptance scenario for the lost-Synchronization incident: pass 1 fails on one hook while the remaining hooks still emit their Synchronization contexts, pass 2 re-emits contexts for every hook. The failing hook sorts first, so the test catches both regressions: a revert of the best-effort aggregation in HandleModuleEnableKubernetesBindings (pass 1 would emit zero contexts) and a non-idempotent shell-operator Enable (pass 2 would emit only the failed hook's context). Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
A Synchronization task that is built but never queued left its binding invisible to SynchronizationState.IsCompleted (vacuously completed) and its kubernetes events locked forever, with only an Error log. - ensure hook queues exist at the start of the QueueSynchronizationTasks phase: queues are normally created in the Startup phase only when DoModuleStartup is set, and a converge restart can displace that task, making every later AddLastTaskToQueue fail deterministically - collect queueing errors and fail the phase instead of skipping the task: the retry rebuilds every Synchronization context because EnableKubernetesBindings is idempotent - log 'append' only for tasks that were actually queued Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
74703bc to
8877274
Compare
Glitchy-Sheep
added a commit
to deckhouse/deckhouse
that referenced
this pull request
Jul 10, 2026
…ifetime fixes Picks up shell-operator 38c7e4f (Error log + binding_monitor_missing_total in SnapshotsFor) and ed5d828 (shared informer factory lifetime owned by the manager + factory_informer_dead_total), and addon-operator 8877274 (rebased onto main past v1.24.0: Synchronization retry test, fail-on-unqueued-sync-task fix, shell-operator pin bump; brings module-sdk v0.12.0 transitively). Root and tools modules together, so 'go generate' stays green. Pseudo-versions to be replaced with release tags once the upstream PRs merge (flant/shell-operator#922, flant/addon-operator#797). Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Release tag for flant/shell-operator#922 (tree-identical to the previously pinned branch pseudo-version): panic isolation in ManagerEventsHandler, idempotent self-healing EnableKubernetesBindings, binding_monitor_missing_total, FactoryStore lifetime owned by the manager + factory_informer_dead_total. The best-effort enable loop (errors.Join) is only safe together with the idempotent EnableKubernetesBindings from this release: on an older shell-operator, pass 1 starts monitors for every successful hook and the retry then hits the alreadyEnabled early-return, which yields zero Synchronization contexts for all of them. Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
8877274 to
39f7473
Compare
ldmonster
approved these changes
Jul 10, 2026
Glitchy-Sheep
added a commit
to deckhouse/deckhouse
that referenced
this pull request
Jul 10, 2026
Release tags for the hook machinery fixes (flant/shell-operator#922, flant/addon-operator#797, both merged): - panic isolation in ManagerEventsHandler - idempotent self-healing EnableKubernetesBindings, Synchronization is re-emitted on every enable call - module hook registration is idempotent and race-free - Synchronization tasks are not silently dropped on queueing errors - shared informer factory lifetime is owned by the kube events manager - new counters: binding_monitor_missing_total, factory_informer_dead_total Root and tools modules together, so 'go generate' stays green. Brings module-sdk v0.12.0 transitively. Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Glitchy-Sheep
added a commit
to deckhouse/deckhouse
that referenced
this pull request
Jul 10, 2026
Release tags for the hook machinery fixes (flant/shell-operator#922, flant/addon-operator#797, both merged): - panic isolation in ManagerEventsHandler - idempotent self-healing EnableKubernetesBindings, Synchronization is re-emitted on every enable call - module hook registration is idempotent and race-free - Synchronization tasks are not silently dropped on queueing errors - shared informer factory lifetime is owned by the kube events manager - new counters: binding_monitor_missing_total, factory_informer_dead_total Root and tools modules together, so 'go generate' stays green. Brings module-sdk v0.12.0 transitively. Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Glitchy-Sheep
added a commit
to deckhouse/deckhouse
that referenced
this pull request
Jul 10, 2026
Release tags for the hook machinery fixes (flant/shell-operator#922, flant/addon-operator#797, both merged): - panic isolation in ManagerEventsHandler - idempotent self-healing EnableKubernetesBindings, Synchronization is re-emitted on every enable call - module hook registration is idempotent and race-free - Synchronization tasks are not silently dropped on queueing errors - shared informer factory lifetime is owned by the kube events manager - new counters: binding_monitor_missing_total, factory_informer_dead_total Root and tools modules together, so 'go generate' stays green. Brings module-sdk v0.12.0 transitively. Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Glitchy-Sheep
added a commit
to deckhouse/deckhouse
that referenced
this pull request
Jul 10, 2026
Release tags for the hook machinery fixes (flant/shell-operator#922, flant/addon-operator#797, both merged): - panic isolation in ManagerEventsHandler - idempotent self-healing EnableKubernetesBindings, Synchronization is re-emitted on every enable call - module hook registration is idempotent and race-free - Synchronization tasks are not silently dropped on queueing errors - shared informer factory lifetime is owned by the kube events manager - new counters: binding_monitor_missing_total, factory_informer_dead_total Root and tools modules together, so 'go generate' stays green. Brings module-sdk v0.12.0 transitively. Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Fixes three bugs in the module hook machinery:
AddHookreplaces the old hook object in all bindings instead of appending a duplicateregistered/controllersReadyflags are read and written under the storage lockRegisterHooksis serialized with a mutexgetHooksreturns a sorted copy instead of the shared sliceHandleModuleEnableKubernetesBindingstries every hook and aggregates errors witherrors.Join, instead of aborting on the first failureEnableKubernetesBindings)QueueSynchronizationTasksphaseAlso bumps shell-operator to the released v1.20.2.
Why we need it
Production incidents:
HookController, the retry appended duplicates next to them, and the kube-events dispatcher crashed on the stale entry.internalvalues, and the module render shrank.Before: one transient failure poisons the hook index or drops the module's Synchronization run for good.
After: a registration retry replaces stale hooks; a partial enable failure is retried, and the retry re-emits Synchronization for every hook.
Notes for the reviewer
alreadyEnabledearly-return and gets zero Synchronization contexts.EnableKubernetesBindings.-race:TestAddHookIsIdempotentPerBindingTestRegistrationRetryLeavesNoOrphanHooksTestControllersReadyFlagIsRaceFreeTestConcurrentRegisterHooksIsSafeTestHandleModuleEnableKubernetesBindings_RetryAfterPartialFailure- pass 1 fails on one hook, the others still emit contexts; pass 2 re-emits for all