New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Log non-graceful termination to /var/log/kube-apiserver/termination.log and stdout #876
Log non-graceful termination to /var/log/kube-apiserver/termination.log and stdout #876
Conversation
cbe2285
to
d5b388f
Compare
d36f601
to
6188db2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd prefer a carry patch on kube-apiserver which is unlikely to conflict and has lower risk of getting some signal handling wrong, but this doesn't look too bad either :)
9730e1e
to
164fe52
Compare
/test e2e-gcp-upgrade |
@p0lyn0mial: The specified target(s) for
Use
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
654d7df
to
57486ec
Compare
/retest |
1 similar comment
/retest |
3817cdb
to
6e96841
Compare
I really don't like nested signal termination in this case either - why wouldn't kube-apiserver summarize failure better on exit to logs? That's why brian and I added fallbacktologsonerror - so that our infra components could log better failures on exit? |
/retest |
Flakes. /retest |
Because we don't have logs. Logging of termination is broken in kubelet since 4.1. Our BZ is open since 4.1 too. Am happy to remove this again as soon as a) kubelet and cri-o start working and doing their job We are spending half of our time hunting issues because we are blind. Waiting is no option. |
/lgtm |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
20 similar comments
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/hold failing
|
openshift/origin#25282 merged. /retest |
/hold cancel |
/retest Please review the full test history for this PR and help us cut down flakes. |
1 similar comment
/retest Please review the full test history for this PR and help us cut down flakes. |
This
NonGracefulTermination
events in the openshift-kube-apiserver namespace on next launch.Container logs alone are not enough because logging during termination is broken, and we lose logs from old pods. So we have to persist the data somewhere on disk.
Etcd is also no option as etcd struggles with the same termination issues and is not reliable.
Depends on openshift/origin#25192.