apiserver still eats panic

[xiang90]

We try to stop everything from eating panics here: #28800.

But API server still eats panics since the HTTP Server catches panics.

See this example:

package main

import (
    "fmt"
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/panic", func(w http.ResponseWriter, r *http.Request) {
        defer func() {
            if r := recover(); r != nil {
                fmt.Println("handle panic")
                panic(r)
            }
        }()

        fmt.Fprintf(w, "panic")
        panic("panic")
    })

    log.Fatal(http.ListenAndServe(":8080", nil))
}

handle panic
2016/08/09 11:12:51 http: panic serving 127.0.0.1:61476: panic
goroutine 20 [running]:
net/http.(*conn).serve.func1(0xc820086200)
    /usr/local/go/src/net/http/server.go:1389 +0xc1
panic(0x28bca0, 0xc82006eee0)
    /usr/local/go/src/runtime/panic.go:426 +0x4e9
main.main.func1.1()
    /Users/xiangli/playground/panic.go:14 +0x121
panic(0x28bca0, 0xc82006eee0)
    /usr/local/go/src/runtime/panic.go:426 +0x4e9
main.main.func1(0x7857a8, 0xc8200756c0, 0xc8200ec1c0)
    /Users/xiangli/playground/panic.go:19 +0xee
net/http.HandlerFunc.ServeHTTP(0x4062c0, 0x7857a8, 0xc8200756c0, 0xc8200ec1c0)
    /usr/local/go/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc820072a80, 0x7857a8, 0xc8200756c0, 0xc8200ec1c0)
    /usr/local/go/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc820086080, 0x7857a8, 0xc8200756c0, 0xc8200ec1c0)
    /usr/local/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc820086200)
    /usr/local/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2137 +0x44e
handle panic
2016/08/09 11:12:56 http: panic serving 127.0.0.1:61477: panic
goroutine 21 [running]:
net/http.(*conn).serve.func1(0xc820086380)
    /usr/local/go/src/net/http/server.go:1389 +0xc1
panic(0x28bca0, 0xc82006ef90)
    /usr/local/go/src/runtime/panic.go:426 +0x4e9
main.main.func1.1()
    /Users/xiangli/playground/panic.go:14 +0x121
panic(0x28bca0, 0xc82006ef90)
    /usr/local/go/src/runtime/panic.go:426 +0x4e9
main.main.func1(0x7857a8, 0xc820075790, 0xc8200ec2a0)
    /Users/xiangli/playground/panic.go:19 +0xee
net/http.HandlerFunc.ServeHTTP(0x4062c0, 0x7857a8, 0xc820075790, 0xc8200ec2a0)
    /usr/local/go/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc820072a80, 0x7857a8, 0xc820075790, 0xc8200ec2a0)
    /usr/local/go/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc820086080, 0x7857a8, 0xc820075790, 0xc8200ec2a0)
    /usr/local/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc820086380)
    /usr/local/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2137 +0x44e
handle panic
2016/08/09 11:12:57 http: panic serving 127.0.0.1:61478: panic
goroutine 22 [running]:
net/http.(*conn).serve.func1(0xc820086480)
    /usr/local/go/src/net/http/server.go:1389 +0xc1
panic(0x28bca0, 0xc82006f020)
    /usr/local/go/src/runtime/panic.go:426 +0x4e9
main.main.func1.1()
    /Users/xiangli/playground/panic.go:14 +0x121
panic(0x28bca0, 0xc82006f020)
    /usr/local/go/src/runtime/panic.go:426 +0x4e9
main.main.func1(0x7857a8, 0xc820075930, 0xc8200ec380)
    /Users/xiangli/playground/panic.go:19 +0xee
net/http.HandlerFunc.ServeHTTP(0x4062c0, 0x7857a8, 0xc820075930, 0xc8200ec380)
    /usr/local/go/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc820072a80, 0x7857a8, 0xc820075930, 0xc8200ec380)
    /usr/local/go/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc820086080, 0x7857a8, 0xc820075930, 0xc8200ec380)
    /usr/local/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc820086480)
    /usr/local/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2137 +0x44e

/cc @lavalamp @krousey

[xiang90]

I would like to get this fixed if everyone agrees that this is an issue we should resolve.

[jayunit100]

Yeah, we're seeing this .

[timothysc]

I turned off a bunch of the panic eating b/c failing fast is a 1st order principle in dist-systems, but "other folks" added it back.

[jayunit100]

yeah, maybe running w/ really-crash=true is the workaround, i suspect the generalized reproduction scenario outside of our systems is slow network, high max inflight, high apiserver capacity : @jeremyeder .

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close