Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cut off connections from kube-aggregator to aggregated apiserver lead to panic in the logs #82591

Closed
sttts opened this issue Sep 11, 2019 · 3 comments · Fixed by #82588
Closed

Cut off connections from kube-aggregator to aggregated apiserver lead to panic in the logs #82591

sttts opened this issue Sep 11, 2019 · 3 comments · Fixed by #82588
Assignees
@liggitt
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Milestone
v1.16

Comments

@sttts
Copy link
Contributor

sttts commented Sep 11, 2019 •
edited

The kube-aggregator uses Golang's reverse proxy code to forward requests to aggregated API server. Whenever such a connection is uncleanly closed, the Golang code panics. The panic is passed through the handler chain down to the panic handler which prints it.

This leads to main panic stack traces in the kube-apiserver logs whenever an aggregated API server terminates, because every long-running connection (e.g. watches) is uncleanly closed leading to that panic.

This applies to 1.16.
@sttts sttts added the kind/bug Categorizes issue or PR as related to a bug. label Sep 11, 2019
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Sep 11, 2019
@sttts sttts added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 11, 2019
@sttts sttts added this to the v1.16 milestone Sep 11, 2019
@kubernetes kubernetes deleted a comment from k8s-ci-robot Sep 11, 2019
@liggitt liggitt mentioned this issue Sep 11, 2019
Merged
@liggitt
Copy link
Member

liggitt commented Sep 11, 2019

/priority critical-urgent

given this is a regression in 1.16 and spams the log badly, making it difficult to see actual errors

@k8s-ci-robot k8s-ci-robot added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Sep 11, 2019
@liggitt liggitt changed the title Cut off connections from kube-aggregator to aggregated apiservrer lead to panic in the logs Cut off connections from kube-aggregator to aggregated apiserver lead to panic in the logs Sep 11, 2019
@liggitt liggitt removed the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Sep 11, 2019
@lachie83
Copy link
Member

@sttts @liggitt as I'm sure you are both already aware that the 1.16.0 cherry-pick deadline is EoD tomorrow. Do you expect to have a PR and associate cherry-pick PR ready to merge in that time frame?

@liggitt
Copy link
Member

liggitt commented Sep 11, 2019

Do you expect to have a PR and associate cherry-pick PR ready to merge in that time frame?

PRs are open/green/approved for master (#82588) and 1.16 (#82593) branches

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liggitt liggitt

Labels
kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Projects
None yet
Milestone
v1.16
Development

Successfully merging a pull request may close this issue.

Propagate and honor http.ErrAbortHandler liggitt/kubernetes
4 participants
@sttts @liggitt @lachie83 @k8s-ci-robot