endpoint: Fix goroutine leak when EP is deleted #13683
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
christarazi
added
area/daemon
maintainer-s-little-helper
bot
added
dont-merge/needs-release-note-label
christarazi
force-pushed
the
pr/christarazi/fix-goroutine-leak-metadata-resolver
branch
from
6f0e342
to
f118e0d
Compare
|
test-me-please |
aanm
approved these changes
Oct 21, 2020
|
test-gke |
The metadata resolver controller can be put in a scenario where it can never succeed if the Cilium K8s pod cache is out-of-sync with the current view of pods in the cluster. The cache can be out-of-sync if there's heavy load on the cluster and especially on the apiserver. This leaves the Cilium pod cache starved. When the pod cache is starved, Cilium may never have an up-to-date view of the pods running in the cluster, and therefore may never be able to resolve labels for pods because the fetch will return "pod not found". Eventually, kubelet (or maybe even the user) will give up on the pod and remove it, thereby Cilium begins removing the endpoint. The controller is stopped, but the goroutine waiting on the `done` channel will never receive, hence the leak. This commit fixes the goroutine leak when the controller never succeeds and therefore, never closes the `done` channel. The fix is to add a `select` statement to also watch the endpoint's `aliveCtx` which is cancelled (and the context's `Done` channel is closed) when the endpoint is deleted. This commit was validated by forcing the metadata resolver to never find a pod (manually hardcode the wrong pod name), and ensuring that the `gops stack` output does not contain an entry like: ``` goroutine 1244259 [chan receive, 1434 minutes]: github.com/cilium/cilium/pkg/endpoint.(*Endpoint).RunMetadataResolver.func1(0xc0055d2a20, 0xc0049ff600, 0xc0055d2ae0, 0x5d) /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1531 +0x34 created by github.com/cilium/cilium/pkg/endpoint.(*Endpoint).RunMetadataResolver /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1530 +0x11e ``` Fixes: cilium#13680 Signed-off-by: Chris Tarazi <chris@isovalent.com>
This controller's run interval is actually unnecessary as this controller is intended to only run once as long as it succeeds. This is evidenced by the fact that we have a `done` channel that's closed when the controller succeeds and a goroutine waiting `done` to be closed to remove the controller, thereby stopping it. In addition, in the very unlikely case that the controller is scheduled again to run _and_ the goroutine waiting for the `done` channel to be closed has not run yet, the controller could panic because it would close the channel twice. This commit prevents that as the interval defaults to 10m if not provided. If the goroutine waiting on the channel doesn't run within 10m, then we very likely have much worse problems at hand. Signed-off-by: Chris Tarazi <chris@isovalent.com>
christarazi
force-pushed
the
pr/christarazi/fix-goroutine-leak-metadata-resolver
branch
from
f118e0d
to
e96f581
Compare
|
test-me-please |
maintainer-s-little-helper
bot
added
the
ready-to-merge
christarazi
deleted the
pr/christarazi/fix-goroutine-leak-metadata-resolver
branch
maintainer-s-little-helper
bot
moved this from Needs backport from master
to Backport pending to v1.7
in 1.7.11
maintainer-s-little-helper
bot
moved this from Needs backport from master
to Backport pending to v1.8
in 1.8.5
maintainer-s-little-helper
bot
moved this from Needs backport from master
to Backport pending to v1.8
in 1.8.5
maintainer-s-little-helper
bot
moved this from Backport pending to v1.7
to Backport done to v1.7
in 1.7.11
maintainer-s-little-helper
bot
moved this from Backport pending to v1.7
to Backport done to v1.7
in 1.7.11
maintainer-s-little-helper
bot
moved this from Needs backport from master
to Backport pending to v1.9
in 1.9.0-rc3
maintainer-s-little-helper
bot
moved this from Needs backport from master
to Backport pending to v1.9
in 1.9.0-rc3
This was referenced Oct 28, 2020
joestringer
moved this from Backport pending to v1.9
to Backport done to v1.9
in 1.9.0-rc3
maintainer-s-little-helper
bot
moved this from Backport pending to v1.8
to Backport done to v1.8
in 1.8.5
maintainer-s-little-helper
bot
moved this from Needs backport from master
to Backport pending to v1.6
in 1.6.13
|
Not backporting this to 1.6 after out-of-band discussion with @christarazi. The build of backport PR #13870 currently fails. and 1.6 will become unsupported with the release of 1.9. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The metadata resolver controller can be put in a scenario where it can
never succeed if the Cilium K8s pod cache is out-of-sync with the
current view of pods in the cluster. The cache can be out-of-sync if
there's heavy load on the cluster and especially on the apiserver. This
leaves the Cilium pod cache starved. When the pod cache is starved,
Cilium may never have an up-to-date view of the pods running in the
cluster, and therefore may never be able to resolve labels for pods
because the fetch will return "pod not found".
Eventually, kubelet (or maybe even the user) will give up on the pod and
remove it, thereby Cilium begins removing the endpoint. The controller
is stopped, but the goroutine waiting on the
donechannel will neverreceive, hence the leak.
This commit fixes the goroutine leak when the controller never succeeds
and therefore, never closes the
donechannel. The fix is to add aselectstatement to also watch the endpoint'saliveCtxwhich iscancelled (and the context's
Donechannel is closed) when the endpointis deleted.
This commit was validated by forcing the metadata resolver to never find
a pod (manually hardcode the wrong pod name), and ensuring that the
gops stackoutput does not contain an entry like:Fixes: #13680