-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
endpoint: Fix goroutine leak when EP is deleted #13683
Merged
joestringer
merged 2 commits into
cilium:master
from
christarazi:pr/christarazi/fix-goroutine-leak-metadata-resolver
Oct 22, 2020
Merged
endpoint: Fix goroutine leak when EP is deleted #13683
joestringer
merged 2 commits into
cilium:master
from
christarazi:pr/christarazi/fix-goroutine-leak-metadata-resolver
Oct 22, 2020
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6f0e342
to
f118e0d
Compare
test-me-please |
joestringer
approved these changes
Oct 21, 2020
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
aanm
approved these changes
Oct 21, 2020
test-gke |
The metadata resolver controller can be put in a scenario where it can never succeed if the Cilium K8s pod cache is out-of-sync with the current view of pods in the cluster. The cache can be out-of-sync if there's heavy load on the cluster and especially on the apiserver. This leaves the Cilium pod cache starved. When the pod cache is starved, Cilium may never have an up-to-date view of the pods running in the cluster, and therefore may never be able to resolve labels for pods because the fetch will return "pod not found". Eventually, kubelet (or maybe even the user) will give up on the pod and remove it, thereby Cilium begins removing the endpoint. The controller is stopped, but the goroutine waiting on the `done` channel will never receive, hence the leak. This commit fixes the goroutine leak when the controller never succeeds and therefore, never closes the `done` channel. The fix is to add a `select` statement to also watch the endpoint's `aliveCtx` which is cancelled (and the context's `Done` channel is closed) when the endpoint is deleted. This commit was validated by forcing the metadata resolver to never find a pod (manually hardcode the wrong pod name), and ensuring that the `gops stack` output does not contain an entry like: ``` goroutine 1244259 [chan receive, 1434 minutes]: github.com/cilium/cilium/pkg/endpoint.(*Endpoint).RunMetadataResolver.func1(0xc0055d2a20, 0xc0049ff600, 0xc0055d2ae0, 0x5d) /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1531 +0x34 created by github.com/cilium/cilium/pkg/endpoint.(*Endpoint).RunMetadataResolver /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1530 +0x11e ``` Fixes: cilium#13680 Signed-off-by: Chris Tarazi <chris@isovalent.com>
This controller's run interval is actually unnecessary as this controller is intended to only run once as long as it succeeds. This is evidenced by the fact that we have a `done` channel that's closed when the controller succeeds and a goroutine waiting `done` to be closed to remove the controller, thereby stopping it. In addition, in the very unlikely case that the controller is scheduled again to run _and_ the goroutine waiting for the `done` channel to be closed has not run yet, the controller could panic because it would close the channel twice. This commit prevents that as the interval defaults to 10m if not provided. If the goroutine waiting on the channel doesn't run within 10m, then we very likely have much worse problems at hand. Signed-off-by: Chris Tarazi <chris@isovalent.com>
f118e0d
to
e96f581
Compare
test-me-please |
This was referenced Oct 28, 2020
Not backporting this to 1.6 after out-of-band discussion with @christarazi. The build of backport PR #13870 currently fails. and 1.6 will become unsupported with the release of 1.9. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area/daemon
Impacts operation of the Cilium daemon.
kind/bug
This is a bug in the Cilium logic.
ready-to-merge
This PR has passed all tests and received consensus from code owners to merge.
release-note/bug
This PR fixes an issue in a previous release of Cilium.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The metadata resolver controller can be put in a scenario where it can
never succeed if the Cilium K8s pod cache is out-of-sync with the
current view of pods in the cluster. The cache can be out-of-sync if
there's heavy load on the cluster and especially on the apiserver. This
leaves the Cilium pod cache starved. When the pod cache is starved,
Cilium may never have an up-to-date view of the pods running in the
cluster, and therefore may never be able to resolve labels for pods
because the fetch will return "pod not found".
Eventually, kubelet (or maybe even the user) will give up on the pod and
remove it, thereby Cilium begins removing the endpoint. The controller
is stopped, but the goroutine waiting on the
done
channel will neverreceive, hence the leak.
This commit fixes the goroutine leak when the controller never succeeds
and therefore, never closes the
done
channel. The fix is to add aselect
statement to also watch the endpoint'saliveCtx
which iscancelled (and the context's
Done
channel is closed) when the endpointis deleted.
This commit was validated by forcing the metadata resolver to never find
a pod (manually hardcode the wrong pod name), and ensuring that the
gops stack
output does not contain an entry like:Fixes: #13680