Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automated cherry pick of #115620: client-go/cache: fix missing delete event on replace (+ #116623) #115900

Merged
merged 6 commits into from Apr 4, 2023
Merged

Automated cherry pick of #115620: client-go/cache: fix missing delete event on replace (+ #116623) #115900

merged 6 commits into from Apr 4, 2023

Conversation

odinuge
Copy link
Member

@odinuge odinuge commented Feb 20, 2023 •
edited

Cherry pick of #115620 on release-1.25.

#115620: client-go/cache: fix missing delete event on replace

For details on the cherry pick process, see the cherry pick requests page.

Fix missing delete events on informer re-lists to ensure all delete events are correctly emitted and using the latest known object state, so that all event handlers and stores always reflect the actual apiserver state as best as possible

@odinuge
client-go/cache: fix missing delete event on replace
bdc4a22 
This fixes a race condition when a "short lived" object
is created and the create event is still present on the queue
when a relist replaces the state. Previously that would lead in the
object being leaked.

The way this could happen is roughly;

1. new Object is added O, agent gets CREATED event for it
2. watch is terminated, and the agent runs a new list, L
3. CREATE event for O is still on the queue to be processed.
4. informer replaces the old data in store with L, and O is not in L
  - Since O is not in the store, and not in the list L, no DELETED event
    is queued
5. CREATE event for O is still on the queue to be processed.
6. CREATE event for O is processed
7. O is <leaked>; its present in the cache but not in k8s.

With this patch, on step 4. above it would create a DELETED event
ensuring that the object will be removed.

Signed-off-by: Odin Ugedal <ougedal@palantir.com>
Signed-off-by: Odin Ugedal <odin@uged.al>
@odinuge
client-go/cache: fix missing delete event on replace without knownObj…
a818874 
…ects

This fixes an issue where a relist could result in a DELETED delta
with an object wrapped in a DeletedFinalStateUnknown object; and then on
the next relist, it would wrap that object inside another
DeletedFinalStateUnknown, leaving the user with a "double" layer
of DeletedFinalStateUnknown's.

Signed-off-by: Odin Ugedal <ougedal@palantir.com>
Signed-off-by: Odin Ugedal <odin@uged.al>
@odinuge
client-go/cache: merge ReplaceMakesDeletionsForObjectsInQueue tests
db451e1 
Signed-off-by: Odin Ugedal <ougedal@palantir.com>
Signed-off-by: Odin Ugedal <odin@uged.al>
@odinuge
client-go/cache: rewrite Replace to check queue first
4283020 
This is useful to both reduce the code complexity, and to ensure clients
get the "newest" version of an object known when its deleted. This is
all best-effort, but for clients it makes more sense giving them the
newest object they observed rather than an old one.

This is especially useful when an object is recreated. eg.

Object A with key K is in the KnownObjects store;
- DELETE delta for A is queued with key K
- CREATE delta for B is queued with key K
- Replace without any object with key K in it.

In this situation its better to create a DELETE delta with
DeletedFinalStateUnknown with B (with this patch), than it is to give
the client an DeletedFinalStateUnknown with A (without this patch).

Signed-off-by: Odin Ugedal <ougedal@palantir.com>
Signed-off-by: Odin Ugedal <odin@uged.al>
@odinuge
client-go/cache: update Replace comment to be more clear
cc675a5 
Since the behavior is now changed, and the old behavior leaked objects,
this adds a new comment about how Replace works.

Signed-off-by: Odin Ugedal <ougedal@palantir.com>
Signed-off-by: Odin Ugedal <odin@uged.al>
@k8s-ci-robot k8s-ci-robot added do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Feb 20, 2023
@k8s-ci-robot k8s-ci-robot added this to the v1.25 milestone Feb 20, 2023
@k8s-ci-robot k8s-ci-robot added the do-not-merge/cherry-pick-not-approved Indicates that a PR is not yet approved to merge into a release branch. label Feb 20, 2023
@k8s-ci-robot
Copy link
Contributor

This cherry pick PR is for a release branch and has not yet been approved by Release Managers.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick, it must first be approved (/lgtm + /approve) by the relevant OWNERS.

AFTER it has been approved by code owners, please leave the following comment on a line by itself, with no leading whitespace: /cc kubernetes/release-managers

(This command will request a cherry pick review from Release Managers and should work for all GitHub users, whether they are members of the Kubernetes GitHub organization or not.)

For details on the patch release process and schedule, see the Patch Releases page.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 20, 2023
@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Feb 20, 2023
@aojea
Copy link
Member

aojea commented Feb 20, 2023

/lgtm
/assign @lavalamp @wojtek-t

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 20, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 529cf854ebd0493b0790ef0d5f9f4bcdf3446da3

@wojtek-t
Copy link
Member

/kind bug
/priority important-soon

/lgtm
/approve

@kubernetes/release-managers - for cherrypick approval

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Feb 20, 2023
@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 21, 2023
@lavalamp
Copy link
Member

/lgtm
/approve

@odinuge
Copy link
Member Author

odinuge commented Mar 1, 2023

Adding a hold while we discuss #115658 (comment).

/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 1, 2023
@Verolop
Copy link

Verolop commented Mar 10, 2023

@lavalamp per @odinuge last comment on #115658 (comment), I'm happy to add this as a cherry pick for the upcoming patches. Can you please remove the hold tag? thanks!

@lavalamp
Copy link
Member

Please add #116623 to this cherry pick and then it should be good to go.

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 17, 2023
@lavalamp @odinuge
Change where transformers are called.
a247e48 
odinuge: sorted out some function signature changes during
cherry-picking that caused conflicts.

(cherry picked from commit e76dff3)
Signed-off-by: Odin Ugedal <odin@uged.al>
@odinuge odinuge force-pushed the automated-cherry-pick-of-#115620-upstream-release-1.25 branch from 876b515 to a247e48 Compare March 17, 2023 14:51
@liggitt
Copy link
Member

liggitt commented Mar 17, 2023

/retest

@odinuge odinuge changed the title Automated cherry pick of #115620: client-go/cache: fix missing delete event on replace Automated cherry pick of #115620: client-go/cache: fix missing delete event on replace (+ #116623) Mar 28, 2023
@odinuge
Copy link
Member Author

odinuge commented Mar 28, 2023

/hold cancel

cc @kubernetes/release-managers for approval

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 28, 2023
@odinuge
Copy link
Member Author

odinuge commented Mar 28, 2023

Mind +1'ing again @lavalap, thanks!

@liggitt
Copy link
Member

liggitt commented Mar 30, 2023

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 30, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 331712b5c8d7ac3cb213329506a8da1d5298a396

@cici37
Copy link
Contributor

cici37 commented Mar 30, 2023

Shouldn't the release note be added together as in original PR?

@k8s-ci-robot k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Mar 31, 2023
xmudrii
xmudrii approved these changes Apr 4, 2023
View reviewed changes
Copy link
Member

@xmudrii xmudrii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For RelEng:
/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lavalamp, odinuge, wojtek-t, xmudrii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@xmudrii xmudrii added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Apr 4, 2023
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/cherry-pick-not-approved Indicates that a PR is not yet approved to merge into a release branch. label Apr 4, 2023
@k8s-ci-robot k8s-ci-robot merged commit 5cd05fa into kubernetes:release-1.25 Apr 4, 2023
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xmudrii xmudrii xmudrii approved these changes

@derekwaynecarr derekwaynecarr Awaiting requested review from derekwaynecarr

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

@aojea aojea Awaiting requested review from aojea

@lavalamp lavalamp Awaiting requested review from lavalamp

@wojtek-t wojtek-t Awaiting requested review from wojtek-t

Assignees

@lavalamp lavalamp

@liggitt liggitt

@aojea aojea

@wojtek-t wojtek-t

@xmudrii xmudrii

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
v1.25
Development

Successfully merging this pull request may close these issues.

None yet
9 participants
@odinuge @k8s-ci-robot @aojea @wojtek-t @cici37 @lavalamp @Verolop @liggitt @xmudrii