Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

In GuaranteedUpdate, retry on any error if we are working with cached data #77619

Merged
merged 1 commit into from May 13, 2019
Merged

In GuaranteedUpdate, retry on any error if we are working with cached data #77619

merged 1 commit into from May 13, 2019

Conversation

caesarxuchao
Copy link
Member

@caesarxuchao caesarxuchao commented May 8, 2019 •
edited by liggitt

/kind bug
/sig api-machinery
/assign @liggitt @jpbetz

Previously, GuaranteedUpdate only retry with refreshed data on "Conflict" error. This is wrong in general, and causes a specific problem we found in #76346, where an object was deleted by the apiserver even if a previous operation had added finalizer to the object. It's because the tryUpdate function returned errDeleteNow based on the stale object in the watch cache. The wrapping GuaranteedUpdate didn't retry with fresh data as errDeleteNow is not a "Conflict" error.

I'm not sure how to add a reliable test to reproduce flakes in #76346. I plan to add a unit test to store_test.go, with a fake watchcache that always returns stale object.

I checked all calls to GaranteedUpdate. We are lucky. The only problematic behavior caused by this bug is that finalizers get ignored.

Fixed a bug in the apiserver storage that could cause just-added finalizers to be ignored on an immediately following delete request, leading to premature deletion.

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. area/apiserver labels May 8, 2019
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 9, 2019
caesarxuchao
caesarxuchao commented May 9, 2019
View reviewed changes
staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store_test.go
return s.Interface.GuaranteedUpdate(ctx, key, ptrToType, ignoreNotFound, preconditions, tryUpdate, s.cachedObj)
}

func TestDeleteWithCachedObject(t *testing.T) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test fails without the fix.

@caesarxuchao
Copy link
Member Author

Added a unit test that mocks the flake we saw in #76346.

@caesarxuchao
Copy link
Member Author

/retest

@caesarxuchao
Copy link
Member Author

/release-note-none

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 9, 2019
@liggitt
Copy link
Member

liggitt commented May 9, 2019

release-note-none

could this have caused bugs previously? At first glance, this seems like something that should be backported, which also indicates it needs a release note

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels May 9, 2019
@caesarxuchao
Copy link
Member Author

Release note added.

I checked all calls to GaranteedUpdate. We are lucky. The only problematic behavior caused by this bug is that finalizers get ignored.

@caesarxuchao
Copy link
Member Author

/retest

@jpbetz
Copy link
Contributor

jpbetz commented May 13, 2019

I checked all calls to GaranteedUpdate. We are lucky. The only problematic behavior caused by this bug is that finalizers get ignored.

Thanks for checking. I found the ways the we're performing reads with GuaranteedUpdate to be generally difficult to reason about and error prone, but maybe that's a topic for another day.

I've walked through the code more carefully and it does fix the issue. I'm in favor of getting it in and backported.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 13, 2019
@liggitt
Copy link
Member

liggitt commented May 13, 2019

/approve

@liggitt
Copy link
Member

liggitt commented May 13, 2019

/priority critical-urgent

@k8s-ci-robot k8s-ci-robot added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels May 13, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: caesarxuchao, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 13, 2019
@caesarxuchao
Copy link
Member Author

/retest

@k8s-ci-robot k8s-ci-robot merged commit ed4c508 into kubernetes:master May 13, 2019
This was referenced May 14, 2019
Merged
Merged
Merged
k8s-ci-robot added a commit that referenced this pull request May 18, 2019
@k8s-ci-robot
Merge pull request #77880 from caesarxuchao/automated-cherry-pick-of-…
3e39ad0 
…#77619-upstream-release-1.12

Automated cherry pick of #77619: In GuaranteedUpdate, retry on any error if we are working
k8s-ci-robot added a commit that referenced this pull request May 21, 2019
@k8s-ci-robot
Merge pull request #77879 from caesarxuchao/automated-cherry-pick-of-…
b458486 
…#77619-upstream-release-1.13

Automated cherry pick of #77619: In GuaranteedUpdate, retry on any error if we are working
k8s-ci-robot added a commit that referenced this pull request May 21, 2019
@k8s-ci-robot
Merge pull request #77875 from caesarxuchao/automated-cherry-pick-of-…
92e692d 
…#77619-upstream-release-1.14

Automated cherry pick of #77619: In GuaranteedUpdate, retry on any error if we are working
@wojtek-t wojtek-t mentioned this pull request Nov 20, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jpbetz jpbetz jpbetz left review comments

@liggitt liggitt liggitt left review comments

@timothysc timothysc Awaiting requested review from timothysc

@wojtek-t wojtek-t Awaiting requested review from wojtek-t

Assignees

@jpbetz jpbetz

@liggitt liggitt

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@caesarxuchao @liggitt @jpbetz @k8s-ci-robot