Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

excessive watch cancellation logging on server shutdown #93450

Closed
liggitt opened this issue Jul 26, 2020 · 4 comments · Fixed by #94105
Closed

excessive watch cancellation logging on server shutdown #93450

liggitt opened this issue Jul 26, 2020 · 4 comments · Fixed by #94105
Assignees
@jingyih
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Milestone
v1.19

Comments

@liggitt
Copy link
Member

liggitt commented Jul 26, 2020

when a server is shut down, all watchers terminate, and are now logging that they are going to send a watch cancel request, and that they failed:

...
I0726 02:59:40.069592   54198 ???:1] sending watch cancel request for closed watcher{watch-id 11 0  <nil>}
W0726 02:59:40.069628   54198 ???:1] failed to send watch cancel request{watch-id 11 0  <nil>} {error 25 0  EOF}
I0726 02:59:40.069770   54198 ???:1] sending watch cancel request for closed watcher{watch-id 11 0  <nil>}
W0726 02:59:40.069786   54198 ???:1] failed to send watch cancel request{watch-id 11 0  <nil>} {error 25 0  EOF}
I0726 02:59:40.069864   54198 ???:1] sending watch cancel request for closed watcher{watch-id 11 0  <nil>}
W0726 02:59:40.069881   54198 ???:1] failed to send watch cancel request{watch-id 11 0  <nil>} {error 25 0  EOF}
I0726 02:59:40.070058   54198 ???:1] sending watch cancel request for closed watcher{watch-id 11 0  <nil>}
W0726 02:59:40.070075   54198 ???:1] failed to send watch cancel request{watch-id 11 0  <nil>} {error 25 0  EOF}
I0726 02:59:40.070236   54198 ???:1] sending watch cancel request for closed watcher{watch-id 11 0  <nil>}
W0726 02:59:40.070254   54198 ???:1] failed to send watch cancel request{watch-id 11 0  <nil>} {error 25 0  EOF}
I0726 02:59:40.070379   54198 ???:1] sending watch cancel request for closed watcher{watch-id 11 0  <nil>}
W0726 02:59:40.070394   54198 ???:1] failed to send watch cancel request{watch-id 11 0  <nil>} {error 25 0  EOF}
...

looks like this was added in etcd in etcd-io/etcd#11850 and picked to 3.4.10 in etcd-io/etcd#12055

/cc @jingyih @jpbetz
@liggitt liggitt added the kind/bug Categorizes issue or PR as related to a bug. label Jul 26, 2020
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jul 26, 2020
@liggitt liggitt changed the title excessive watch logging excessive watch cancellation logging on server shutdown Jul 26, 2020
@liggitt
Copy link
Member Author

liggitt commented Jul 26, 2020

/milestone v1.19
/sig api-machinery

@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 26, 2020
@k8s-ci-robot k8s-ci-robot added this to the v1.19 milestone Jul 26, 2020
@jingyih jingyih mentioned this issue Jul 29, 2020
Merged
@liggitt
Copy link
Member Author

liggitt commented Aug 3, 2020

Pending pick to etcd 3.4.x in etcd-io/etcd#12189

@jingyih jingyih mentioned this issue Aug 13, 2020
Merged
@liggitt liggitt added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Aug 14, 2020
@liggitt
Copy link
Member Author

liggitt commented Aug 18, 2020

pending 3.4.11 release (should be ~today)

delta from current level of 3.4.10: etcd-io/etcd@v3.4.10...release-3.4

@neolit123 neolit123 mentioned this issue Aug 19, 2020
Closed
@liggitt liggitt mentioned this issue Aug 19, 2020
Merged
@liggitt
Copy link
Member Author

liggitt commented Aug 19, 2020

fixed in #94105

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jingyih jingyih

Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Projects
None yet
Milestone
v1.19
Development

Successfully merging a pull request may close this issue.

Update to etcd client version 3.4.12 liggitt/kubernetes
3 participants
@liggitt @k8s-ci-robot @jingyih