New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Propagate panics up handler chain #71076

Merged
merged 1 commit into from Nov 16, 2018

Conversation

@liggitt
Copy link
Member

liggitt commented Nov 15, 2018

What type of PR is this?
/kind bug

What this PR does / why we need it:

Fixes panic propagation in finishRequest. finishRequest incorrectly assumed that HandleCrash was the correct way to recover a panic in order to propagate it up the handler chain. In actuality, HandleCrash simply logs and repanics, which causes the process to exit in finishRequest, which means we don't get the full call stack. We want to capture the panic reason/stack, and propagate it upward so the full stack is logged at the top of the handler chain.

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

apiserver: fixes handling and logging of panics in REST handlers to prevent crashes
@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Nov 15, 2018

/assign lavalamp sttts

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Nov 15, 2018

/priority critical-urgent
/sig api-machinery

@liggitt liggitt force-pushed the liggitt:preserve-stack branch from e295a2f to 0b39f63 Nov 15, 2018

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 15, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@liggitt liggitt force-pushed the liggitt:preserve-stack branch from 0b39f63 to 57e7e3c Nov 15, 2018

const size = 64 << 10
buf := make([]byte, size)
buf = buf[:goruntime.Stack(buf, false)]
panicReason = fmt.Sprintf("%v\n%s", panicReason, string(buf))

This comment has been minimized.

@sttts

sttts Nov 15, 2018

Contributor

trim trailing newline

This comment has been minimized.

@sttts

sttts Nov 15, 2018

Contributor

Makes concatenation of further backtraces cleaner

This comment has been minimized.

@liggitt
@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Nov 15, 2018

/milestone v1.13

@sttts

This comment has been minimized.

Copy link
Contributor

sttts commented Nov 15, 2018

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label Nov 15, 2018

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Nov 15, 2018

/retest

1 similar comment
@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Nov 15, 2018

/retest

@fedebongio

This comment has been minimized.

Copy link
Contributor

fedebongio commented Nov 15, 2018

/cc @roycaihw

@k8s-ci-robot k8s-ci-robot requested a review from roycaihw Nov 15, 2018

@k8s-ci-robot k8s-ci-robot merged commit f1e4ec8 into kubernetes:master Nov 16, 2018

18 checks passed

cla/linuxfoundation liggitt authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Skipped
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-e2e-kubeadm-gce Skipped
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details

@liggitt liggitt deleted the liggitt:preserve-stack branch Nov 16, 2018

k8s-ci-robot added a commit that referenced this pull request Nov 28, 2018

Merge pull request #71160 from liggitt/automated-cherry-pick-of-#71067-
…#71076-upstream-release-1.12

Automated cherry pick of #71067: apiserver: in timeout_test separate out handler #71076: apiserver: propagate panics from REST handlers correctly

k8s-ci-robot added a commit that referenced this pull request Nov 30, 2018

Merge pull request #71163 from liggitt/automated-cherry-pick-of-#71067-
…#71076-upstream-release-1.10

Automated cherry pick of #71067: apiserver: in timeout_test separate out handler #71076: apiserver: propagate panics from REST handlers correctly

k8s-ci-robot added a commit that referenced this pull request Dec 3, 2018

Merge pull request #71161 from liggitt/automated-cherry-pick-of-#71067-
…#71076-upstream-release-1.11

Automated cherry pick of #71067: apiserver: in timeout_test separate out handler #71076: apiserver: propagate panics from REST handlers correctly
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment