operator: fix deadlock when running in kvstore mode #24631
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When running in kvstore mode, the start hook of the identity GC cell blocks until the kvstore client has been initialized, which is performed by the legacyCell start hook. Given that the identity GC cell was registered first, and there are no explicit dependencies among the two, its start hook was also executed first, causing a deadlock. This commit changes the order in which the cells are registered as a workaround, until the kvstore is refactored into a proper cell. The current hooks execution order is the following: function="gops.registerGopsHooks.func1 (cell.go:44)" function="client.(*compositeClientset).onStart" function="cmd.registerOperatorHooks.func1 (root.go:142)" function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/apis/cilium.io/v2alpha1.CiliumLoadBalancerIPPool].Start" function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/slim/k8s/api/core/v1.Service].Start" function="*lbipam.LBIPAM.Start" function="*resource.resource[*k8s.io/api/core/v1.Node].Start" function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/apis/cilium.io/v2.CiliumNode].Start" function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/slim/k8s/api/core/v1.Namespace].Start" function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/apis/cilium.io/v2.CiliumIdentity].Start" function="cmd.(*legacyOnLeader).onStart" function="identitygc.registerGC.func1 (gc.go:107)" Fixes: b115951 ("operator: Refactor cilium identities GC to a cell") Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>
giorio94
added
kind/bug
aanm
approved these changes
Mar 29, 2023
No, since the PR introducing that cell (#22892) was not backported to 1.13. |
|
/test |
aanm
added
release-note/misc
squeed
approved these changes
Mar 29, 2023
|
/ci-gke |
|
/ci-multicluster |
|
/ci-gke |
|
/ci-multicluster |
|
/ci-awscni |
|
All previous failures were due to the GitHub actions issues of yesterday |
|
/test-runtime Hit known flake #23495 |
|
/test-1.25-4.19 Hit known flake: #23236 |
|
/test-1.24-5.4 Hit new flake #24643 Job 'Cilium-PR-K8s-1.24-kernel-5.4' failed: Click to show.Test NameFailure OutputIf it is a flake and a GitHub issue doesn't already exist to track it, comment |
|
/test-1.24-5.4 Hit known flake #16122 |
|
/test-1.24-5.4 Hit known flake #24573 |
maintainer-s-little-helper
bot
added
the
ready-to-merge
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When running in kvstore mode, the start hook of the identity GC cell blocks until the kvstore client has been initialized, which is performed by the legacyCell start hook. Given that the identity GC cell was registered first, and there are no explicit dependencies among the two, its start hook was also executed first, causing a deadlock.
This PR changes the order in which the cells are registered as a workaround, until the kvstore is refactored into a proper cell.
No backport is required, since the original commit is not present in any stable branch.