Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

k8s/watchers: Fix race condition in init functions #23170

Merged

Conversation

christarazi
Copy link
Member

@christarazi christarazi commented Jan 18, 2023

Use an atomic variable as the previous boolean was being passed around
to different goroutines without any synchronization mechanism.

Fixes: b4335ee ("k8s: Introduce shared resources")
Fixes: ee2ccda ("k8s: Add shared resource for namespaces")
Fixes: #23166

Signed-off-by: Chris Tarazi chris@isovalent.com

@christarazi christarazi added sig/k8s Impacts the kubernetes API, or kubernetes -> cilium internals translation layers. release-note/misc This PR makes changes that have no direct user impact. labels Jan 18, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot added dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. and removed dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. labels Jan 18, 2023
@github-actions github-actions bot added the kind/community-contribution This was a contribution made by a community member. label Jan 18, 2023
@christarazi christarazi added kind/bug This is a bug in the Cilium logic. kind/regression This functionality worked fine before, but was broken in a newer release of Cilium. release-blocker/1.13 This issue will prevent the release of the next version of Cilium. needs-backport/1.13 This PR / issue needs backporting to the v1.13 branch and removed kind/community-contribution This was a contribution made by a community member. labels Jan 18, 2023
@christarazi
Copy link
Member Author

/test

@christarazi
Copy link
Member Author

/test-4.19-race

@christarazi
Copy link
Member Author

/test-4.9-race

@christarazi
Copy link
Member Author

Race pipeline tests failed because they found another very similar race condition, pushing a new fix for it too.

@christarazi christarazi force-pushed the pr/christarazi/fix-k8s-race-cond branch from 156a53e to 7b61112 Compare January 18, 2023 19:41
@christarazi
Copy link
Member Author

/test

@christarazi
Copy link
Member Author

/test-4.19-race

@christarazi
Copy link
Member Author

/test-4.9-race

@christarazi christarazi changed the title k8s/watchers: Fix race condition in service init k8s/watchers: Fix race condition in init functions Jan 18, 2023
@christarazi
Copy link
Member Author

Found another instance of the same race condition issue, pushing new fix.

@christarazi christarazi force-pushed the pr/christarazi/fix-k8s-race-cond branch from 7b61112 to 10306fd Compare January 18, 2023 21:53
Use an atomic variable as the previous boolean was being passed around
to different goroutines without any synchronization mechanism.

Fixes: b4335ee ("k8s: Introduce shared resources")
Fixes: ee2ccda ("k8s: Add shared resource for namespaces")
Fixes: cilium#23166

Signed-off-by: Chris Tarazi <chris@isovalent.com>
@christarazi christarazi force-pushed the pr/christarazi/fix-k8s-race-cond branch from 10306fd to c66cec7 Compare January 18, 2023 21:54
@christarazi
Copy link
Member Author

/test

@christarazi
Copy link
Member Author

/test-4.9-race

@christarazi
Copy link
Member Author

/test-4.19-race

@christarazi
Copy link
Member Author

/test

@christarazi christarazi marked this pull request as ready for review January 19, 2023 01:28
@christarazi christarazi requested a review from a team as a code owner January 19, 2023 01:28
@christarazi
Copy link
Member Author

CI triage:

Previously, the race pipeline jobs would fail constantly, but now are passing which proves that the data race fix is working as intended.

@@ -24,12 +25,13 @@ import (
func (k *K8sWatcher) namespacesInit() {
apiGroup := k8sAPIGroupNamespaceV1Core

synced := false
var synced atomic.Bool
synced.Store(false)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is probably redundant, zero value should be false

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I opted for explicitness so that when someone fetches all references of the variable, they see an explicit "false" (next to an explicit "true"). It makes it slightly easier to tell how this variable is used.

@christarazi
Copy link
Member Author

Marking ready to merge as we have approving reviews and CI passing except for known flakes.

@christarazi christarazi added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Jan 19, 2023
@joestringer joestringer merged commit ba68a26 into cilium:master Jan 19, 2023
@christarazi christarazi deleted the pr/christarazi/fix-k8s-race-cond branch January 19, 2023 21:50
@aanm aanm mentioned this pull request Jan 23, 2023
19 tasks
@christarazi christarazi removed needs-backport/1.13 This PR / issue needs backporting to the v1.13 branch release-blocker/1.13 This issue will prevent the release of the next version of Cilium. labels Jan 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug This is a bug in the Cilium logic. kind/regression This functionality worked fine before, but was broken in a newer release of Cilium. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/misc This PR makes changes that have no direct user impact. sig/k8s Impacts the kubernetes API, or kubernetes -> cilium internals translation layers.
Projects
None yet
3 participants