New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Terminate custom resource watches when storage is destroyed #78029
Conversation
/priority critical-urgent this results in hung watch clients with O(hour) timeouts when CRDs change |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: liggitt The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go
Outdated
Show resolved
Hide resolved
one nit. Otherwise lgtm. |
/lgtm |
Impacted how far back? |
Watch cache at least back to 1.12 (I'd expect even further). Custom resources as far back as they used the watch cache (not sure how far that is) |
New changes are detected. LGTM label has been removed. |
fixed gofmt issue |
Looks right. Thanks for finding and fixing. /lgtm |
/lgtm |
…9-upstream-release-1.12 Automated cherry pick of #78029: Terminate watchers when watch cache is destroyed
…9-upstream-release-1.14 Automated cherry pick of #78029: Terminate watchers when watch cache is destroyed
…9-upstream-release-1.13 Automated cherry pick of #78029: Terminate watchers when watch cache is destroyed
What type of PR is this?
/kind bug
What this PR does / why we need it:
When a watch cache is stopped, drop all active watchers.
For normal resources, this only happens on apiserver shutdown. For custom resources, this happens whenever a CRD spec is changed, which currently strands active watchers.
Which issue(s) this PR fixes:
Fixes #74105
Fixes #71138
Does this PR introduce a user-facing change?:
/sig api-machinery
/cc @jpbetz @sttts