Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Error when etcd3 watch finds delete event with nil prevKV #76675
What type of PR is this?
What this PR does / why we need it:
See #76624 (comment), taking half of that suggestion (for now), which is to end the watch when a delete event does not have a prevKV
Does this PR introduce a user-facing change?:
Hi @ryanmcnamara. Thanks for your PR.
I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with
Once the patch is verified, the new status will be reflected by the
I understand the commands that are listed here.
Apr 16, 2019
I'll be testing this change out on our stack that initially ran into #76624 using this branch: https://github.com/ryanmcnamara/kubernetes/tree/rm/error-on-nil-event-demo
@fedebongio: GitHub didn't allow me to request PR reviews from the following users: jingyih.
Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.
The event started exactly when the kube controller manager leader switched (this happens at 1:55). The apiserver it's now hitting is the impacted one. That's how this issue has always reproduced.
Looking at etcd logs and metrics I don't see anything interesting.
Each time this happens I notice load going quite high on the master host, but only around
I don't really think this is the issue though, because the load stays well below
I can take another look at the logs and see if I find anything
Do you want to continue debugging on the issue? Seems like this pr is useful regardless of what's causing the delay in events.
Apr 26, 2019
Apr 26, 2019
[APPROVALNOTIFIER] This PR is APPROVED
The full list of commands accepted by this bot can be found here.
The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing