Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: TimeoutHandler hides panic locations #27375

Open
pvhau opened this issue Aug 30, 2018 · 3 comments

Comments

@pvhau
Copy link

@pvhau pvhau commented Aug 30, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

v1.10.3

Does this issue reproduce with the latest release?

Not sure, I am finding a solution with my current version

What operating system and processor architecture are you using (go env)?

OS: linux, arch: amd64

What did you do?

This is a demo


import (
	"log"
	"net/http"
	"time"
)

type sampleHandler string
func (s sampleHandler) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
	var mm interface{}
	log.Print("This will be panic")
	log.Print(mm.(string))
}

func main() {
	go func() {
		http.ListenAndServe("localhost:8888", http.TimeoutHandler(sampleHandler("sample"), time.Second*10, "Timeout"))
	}()

	time.Sleep(time.Second * 2)
	http.Get("http://localhost:8888/")
}

What did you expect to see?

I expect to see stack trace point to where cause panic (log.Print(mm.(string))) so that I can debug app easily

What did you see instead?

Stack trace point me panic(p) statement in net/http.timeoutHandler (go/src/net/http/server.go:3144). I know it make sense because that is the place that call panic. Are there any good solution to passing stack trace from handler inside TimeoutHandler?

@pvhau pvhau changed the title Passing panic stack trace inside TimeoutHandler Passing panic's stacktrace inside TimeoutHandler Aug 30, 2018
@FiloSottile FiloSottile added this to the Unplanned milestone Aug 31, 2018
@FiloSottile FiloSottile changed the title Passing panic's stacktrace inside TimeoutHandler net/http: TimeoutHandler hides panic locations Aug 31, 2018
@FiloSottile

This comment has been minimized.

Copy link
Member

@FiloSottile FiloSottile commented Aug 31, 2018

/cc @bradfitz

@slon

This comment has been minimized.

Copy link
Contributor

@slon slon commented Oct 11, 2019

Hello,

I would like to contribute fix for this issue.

I intend to change role of the two goroutines in net/http.(*timeoutHandler).ServeHTTP. That way, user handler panicking is still happening on the main goroutine calling the handler and stack trace is preserved for all defer-ed functions higher in the stack.

@bradfitz is this fix OK?

@pam4

This comment has been minimized.

Copy link

@pam4 pam4 commented Oct 12, 2019

Running the inner handler in the main goroutine would also solve more serious problems like #34608.
But currently TimeoutHandler.ServeHTTP doesn't wait for the inner handler to complete in case of timeout, which means that a "timed out" response is never late, even if the inner handler is taking a long time.
By switching goroutines we cannot prevent the "timed out" response to be delayed by the inner handler (even if we write it as soon as possible, Flush is not guaranteed, and the response is not complete until TimeoutHandler.ServeHTTP returns).
So if someone's inner handler runs too long after timeout, this change would break them.

On the other hand, the current situation is quite messy: wrong panic location, panic disappearing after timeout, and broken (racy) Push.
I suspect TimeoutHandler has a very limited user base, and the core team doesn't seem much interested either.
I would like to hear from someone who is actually using TimeoutHandler whether the proposed change is acceptable to them.

To be clear, I'm thinking about a change of this kind in TimeoutHandler.ServeHTTP (untested!):

go func() {
    <-ctx.Done()
    tw.mu.Lock()
    defer tw.mu.Unlock()
    tw.timedOut = true
}
h.handler.ServeHTTP(tw, r)
tw.mu.Lock()
defer tw.mu.Unlock()
if tw.timedOut {
    w.WriteHeader(StatusServiceUnavailable)
    io.WriteString(w, h.errorBody())
    return
}
dst := w.Header()
for k, vv := range tw.h {
    dst[k] = vv
}
if !tw.wroteHeader {
    tw.code = StatusOK
}
w.WriteHeader(tw.code)
w.Write(tw.wbuf.Bytes())

Writing the "timed out" response earlier (in the spawned goroutine) won't make any difference, because of buffering, and because there's no way (that I know of) to tell the server "I'm done" without returning from TimeoutHandler.ServeHTTP.
Also, it's good to keep the spawned goroutine as simple as possible, unless we recover panic in there too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.