Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Retry when recording an EventSerie results in an AlreadyExist error #114237

Merged
merged 1 commit into from Jan 2, 2023
Merged

Retry when recording an EventSerie results in an AlreadyExist error #114237

merged 1 commit into from Jan 2, 2023

Conversation

dgrisonnet
Copy link
Member

What type of PR is this?

/kind bug

What this PR does / why we need it:

When attempting to record a new Event and a new Serie on the apiserver
at the same time, the patch of the Serie might happen before the Event
is actually created. In that case, we handle the error and try to create
the Event. But the Event might be created during that period of time and
it is treated as an error today. So in order to handle that scenario, we
need to retry when a Create call for a Serie results in an AlreadyExist
error.

Special notes for your reviewer:

This issue was highlighted when fixing a data race with a similar scenario and writing a test for it in: #114236.

The test testing this particular scenario was written in the PR above and we can't add it here since it requires the bug fix from the other PR. Also, this would need to go in before the other PR since the test fails without it.

Does this PR introduce a user-facing change?

Fix a bug where when emitting similar Events consecutively, some were rejected by the apiserver.

/cc @wojtek-t

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Dec 1, 2022
@k8s-ci-robot
Copy link
Contributor

Please note that we're already in Test Freeze for the release-1.26 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.26.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Thu Dec 1 09:40:53 UTC 2022.

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Dec 1, 2022
@dgrisonnet dgrisonnet mentioned this pull request Dec 1, 2022
Merged
@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Dec 1, 2022
@fedebongio
Copy link
Contributor

/assign @yliaog
/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 1, 2022
wojtek-t
wojtek-t reviewed Dec 2, 2022
View reviewed changes
staging/src/k8s.io/client-go/tools/events/event_broadcaster.go
// Event is emitted twice in a very short period of time.
if isEventSeries {
return nil, true
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change LGTM. Is it possible add some simple unit test for it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added one in the latest update PTAL

@wojtek-t wojtek-t self-assigned this Dec 2, 2022
@dgrisonnet
tools/events: retry on AlreadyExist for Series
2f83117 
When attempting to record a new Event and a new Serie on the apiserver
at the same time, the patch of the Serie might happen before the Event
is actually created. In that case, we handle the error and try to create
the Event. But the Event might be created during that period of time and
it is treated as an error today. So in order to handle that scenario, we
need to retry when a Create call for a Serie results in an AlreadyExist
error.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Dec 2, 2022
wojtek-t
wojtek-t reviewed Jan 2, 2023
View reviewed changes
staging/src/k8s.io/client-go/tools/events/event_broadcaster_test.go
isomorphicEvent,
},
expectedRecordedEvent: isomorphicEvent,
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome - great tests

@wojtek-t
Copy link
Member

wojtek-t commented Jan 2, 2023

/lgtm
/approve

Thanks!

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 2, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 3e739f22bae4f2df056cf47c71a651c91efce743

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgrisonnet, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@dgrisonnet
Copy link
Member Author

/retest

@k8s-ci-robot k8s-ci-robot merged commit 2b6c842 into kubernetes:master Jan 2, 2023
@k8s-ci-robot k8s-ci-robot added this to the v1.27 milestone Jan 2, 2023
This was referenced Jul 17, 2023
Merged
Merged
k8s-ci-robot added a commit that referenced this pull request Aug 2, 2023
@k8s-ci-robot
Merge pull request #119376 from dgrisonnet/automated-cherry-pick-of-#…
c242e33 
…114237-#114236-#112334-upstream-release-1.25

Automated cherry pick of #114237: tools/events: retry on AlreadyExist for Series
#114236: tools/events: fix data race when emitting series
#112334: events: fix EventSeries starting count discrepancy
k8s-ci-robot added a commit that referenced this pull request Aug 2, 2023
@k8s-ci-robot
Merge pull request #119375 from dgrisonnet/automated-cherry-pick-of-#…
694c7d3 
…114237-#114236-#112334-upstream-release-1.26

Automated cherry pick of #114237: tools/events: retry on AlreadyExist for Series
#114236: tools/events: fix data race when emitting series
#112334: events: fix EventSeries starting count discrepancy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wojtek-t wojtek-t wojtek-t left review comments

Assignees

@wojtek-t wojtek-t

@yliaog yliaog

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
v1.27
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@dgrisonnet @k8s-ci-robot @fedebongio @wojtek-t @yliaog