Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terminate custom resource watches when storage is destroyed #78029

Merged
merged 2 commits into from May 18, 2019

Conversation

@liggitt
Copy link
Member

commented May 17, 2019

What type of PR is this?
/kind bug

What this PR does / why we need it:
When a watch cache is stopped, drop all active watchers.

For normal resources, this only happens on apiserver shutdown. For custom resources, this happens whenever a CRD spec is changed, which currently strands active watchers.

Which issue(s) this PR fixes:
Fixes #74105
Fixes #71138

Does this PR introduce a user-facing change?:

Active watches of custom resources now terminate properly if the CRD is modified.

/sig api-machinery
/cc @jpbetz @sttts

@liggitt

This comment has been minimized.

Copy link
Member Author

commented May 17, 2019

/priority critical-urgent

this results in hung watch clients with O(hour) timeouts when CRDs change

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented May 17, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@liggitt liggitt changed the title Terminate CRD watches when storage is destroyed Terminate custom resource watches when storage is destroyed May 17, 2019

@liggitt liggitt added this to Required for GA, in progress in Custom Resource Definitions May 17, 2019

@sttts

This comment has been minimized.

Copy link
Contributor

commented May 17, 2019

one nit.

Otherwise lgtm.

@liggitt liggitt force-pushed the liggitt:crd-watch branch from dc38a95 to 47ade3b May 17, 2019

@sttts

This comment has been minimized.

Copy link
Contributor

commented May 17, 2019

/lgtm

@smarterclayton

This comment has been minimized.

Copy link
Contributor

commented May 17, 2019

Impacted how far back?

@liggitt

This comment has been minimized.

Copy link
Member Author

commented May 17, 2019

Impacted how far back?

Watch cache at least back to 1.12 (I'd expect even further). Custom resources as far back as they used the watch cache (not sure how far that is)

@liggitt liggitt force-pushed the liggitt:crd-watch branch from 47ade3b to ea46423 May 17, 2019

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented May 17, 2019

New changes are detected. LGTM label has been removed.

@k8s-ci-robot k8s-ci-robot removed the lgtm label May 17, 2019

@liggitt

This comment has been minimized.

Copy link
Member Author

commented May 17, 2019

fixed gofmt issue

@liggitt liggitt added the lgtm label May 17, 2019

@jpbetz

This comment has been minimized.

Copy link
Contributor

commented May 17, 2019

Looks right. Thanks for finding and fixing.

/lgtm

@yliaog

This comment has been minimized.

Copy link
Contributor

commented May 17, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot merged commit 0f8009b into kubernetes:master May 18, 2019

20 checks passed

cla/linuxfoundation liggitt authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Skipped.
pull-kubernetes-cross Skipped.
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-csi-serial Skipped.
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gce-storage-slow Skipped.
pull-kubernetes-godeps Skipped.
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details

@liggitt liggitt moved this from Required for GA, in progress to Complete in Custom Resource Definitions May 20, 2019

@liggitt liggitt deleted the liggitt:crd-watch branch May 20, 2019

k8s-ci-robot added a commit that referenced this pull request May 23, 2019
Merge pull request #78036 from liggitt/automated-cherry-pick-of-#7802…
…9-upstream-release-1.12

Automated cherry pick of #78029: Terminate watchers when watch cache is destroyed
k8s-ci-robot added a commit that referenced this pull request May 31, 2019
Merge pull request #78034 from liggitt/automated-cherry-pick-of-#7802…
…9-upstream-release-1.14

Automated cherry pick of #78029: Terminate watchers when watch cache is destroyed
k8s-ci-robot added a commit that referenced this pull request May 31, 2019
Merge pull request #78035 from liggitt/automated-cherry-pick-of-#7802…
…9-upstream-release-1.13

Automated cherry pick of #78029: Terminate watchers when watch cache is destroyed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.