Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: no way of manipulating timeouts in Handler #16100

FiloSottile opened this issue Jun 17, 2016 · 59 comments

net/http: no way of manipulating timeouts in Handler #16100

FiloSottile opened this issue Jun 17, 2016 · 59 comments


Copy link

@FiloSottile FiloSottile commented Jun 17, 2016

A Handler has no way of changing the underlying connection Deadline, since it has no access to the net.Conn (except by maintaining a map from RemoteAddr to net.Conn via Server.ConnState, but it's more than anyone should need to do). Moreover, it can't implement a timeout itself because the Close method of the ResponseWriter implementation is not documented to unblock concurrent Writes.

This means that if the server has a WriteTimeout, the connection has a definite lifespan, and streaming is impossible. So servers with any streaming endpoints are forced not to implement timeouts at all on the entire Server.

A possible solution might be to expose the net.Conn in the Context. Another could be to allow interface upgrades to the SetDeadline methods on ResponseWriter. Yet another would be to make (*response).Close unblock (*response).Write.

@ianlancetaylor ianlancetaylor added this to the Go1.8 milestone Jun 17, 2016
Copy link

@elithrar elithrar commented Jun 24, 2016

Given that we store the *http.Server in the request context, making net.Conn available in the context via (e.g.) ConnContextKey could be an option. This could be opt-in via a field on the http.Server as stuffing the request context with things by default is not ideal.

Copy link

@noblehng noblehng commented Jul 1, 2016

I think this can be done with providing a custom net.Listener to (*http.Server).Serve.

That is embedding a *net.TCPListener and overwriting the Accept method, which return a custom net.Conn. The custom net.Conn will embed a *net.TCPConn and overwrite the Write method.

The overwritten Write method could reset the write deadline on every write, or use a atomic counter to reset the write deadline on some numbers/bytes of consecutively write. But for truly on demand write deadline resetting, one still need some way to do that on the higer level handler side.

Since a http/2 connection can do multiplexing, it would be helpful to have a set of timeouts for individual stream. When a stream hang on client, we could use those timeouts to release resources associated with that stream, this is not possible with setting deadlines on the lower level underlying connection.

Copy link

@bradfitz bradfitz commented Sep 1, 2016

@FiloSottile, we won't be exposing net.Conn to handlers, or let users explicitly set deadlines on conns. All public APIs need to consider both HTTP/1 and HTTP/2.

You propose many solutions, but I'd like to get a clear statement of the problem first. Even the title of this bug seems like a description of a lack of solution, rather than a problem that's not solvable.

I agree that the WriteTimeout is ill-specified. See also my comment about ReadTimeout here: #16958 (comment)

You allude to your original problem here:

So servers with any streaming endpoints are forced not to implement timeouts at all on the entire Server.

So you have an infinite stream, and you want to forcibly abort that stream if the user isn't reading fast enough? In HTTP/1, that means closing the TCP connection. In HTTP/2, that means sending a RST_STREAM.

I guess we need to define WriteTimeout before we make progress on this bug.

What do you think WriteTimeout should mean?

Copy link

@thincal thincal commented Sep 29, 2016

Why not provide a timeout version of Read/Write ? Is there any reason ?

Request.Body.TimedRead(p []byte, timeout time.Duration)
Request.Body.TimedWrite(p []byte, timeout time.Duration)

My concrete case is:

case1 about read

client send data via. a POST API provided by server, but if the network disconnected during the server read request data via. request.Body.Read, the server will be blocked at Body.Read forever.

case2 about write

similar that it's server write data back to client.

So I need a timeout mechanism that server can be unblocked from this Read/Write.
This is very intuitive requirement, might be there are other solution already ?


Copy link

@bradfitz bradfitz commented Oct 22, 2016

What if the mechanism to abort blocked read or write I/O is to just exit from the ServeHTTP function?

So then handlers can do their own timers before reads/writes and cancel or extend them as necessary. That implies that all such reads & writes would need to be done in a separate goroutine. And then if they're too slow and the timer fires before the hander's I/O goroutine completes, just return from ServeHTTP and we'd either kill the TCP connection (for http1) or do a RST_STREAM (for http2).

I suppose then the problem is there's a small window where the I/O might've completed just in the window between the timer firing and the ServeHTTP code exiting, so then once the http package sees ServeHTTP is done, it may not see any calls still in Read or Write, and might then not fail the connection as intended.

That suggests we need some sort of explicit means of aborting a response. http1 has that with Hijacker.

I've recommended in the past that people just panic, since the http package has that weird feature where it recovers panics.

Would that work? Do all your I/O in separate goroutines, and panic if they take too long?

We might get data races with http1 and/or http2 with that, but I could probably make it work if you like the idea.

Copy link

@gopherbot gopherbot commented Oct 25, 2016

CL mentions this issue.

gopherbot pushed a commit that referenced this issue Oct 26, 2016

Updates #14204
Updates #16450
Updates #16100

Change-Id: Ic283bcec008a8e0bfbcfd8531d30fffe71052531
Reviewed-by: Tom Bergan <>
Reviewed-by: Brad Fitzpatrick <>
Run-TryBot: Brad Fitzpatrick <>
TryBot-Result: Gobot Gobot <>
Copy link

@bradfitz bradfitz commented Nov 1, 2016

Deferring to Go 1.9 due to lack of reply.

@bradfitz bradfitz modified the milestones: Go1.9, Go1.8 Nov 1, 2016
Copy link

@c4milo c4milo commented Nov 1, 2016

@bradfitz I faced a similar problem when I started with Go. I had a service in Node using HTTP 1.1 with chunked transfer encoding, it implemented bidirectional streaming over HTTP 1.1. On each write or read, the deadline/timeout would get reset in order for the connection to remain opened. It worked on Node.js which uses libuv. This predated websockets and avoided long polling. It is a little known technique but is somewhat mentioned in (numeral 3). I also remember Twitter's streaming API using it. Anyway, migrating my Node service to Go was not possible because of the use of absolute timeouts/deadlines.

So, I guess @FiloSottile is referring to a similar case. At least in my particular scenario, what I wanted to do was to be able to reset the connection's write and read timeout/deadline so that the connection remained open but still got closed if it became idle.

Copy link

@drakkan drakkan commented Nov 2, 2016

A timeout that restart on each write or read would be useful for my use case too and if implemented I could rewrite in go a legacy c++ application.

If the timeout could be set per request and not only globally would be a good plus

Copy link
Member Author

@FiloSottile FiloSottile commented Dec 15, 2016

@bradfitz Sorry for not following up on this one, it turned out to be more nuanced than I thought (HTTP/2, brain, HTTP/2 exists) and didn't have the time to look at it further.

Doing I/O in a goroutine instead of the timer sounds upside-down, but I don't have specific points to make against it. What would make more sense to me would be a way to cancel the whole thing, which would make Body.Read/ResponseWriter.Write return an error that would then be handled normally.

Question, without such a mechanism, how is the user supposed to timeout a read when using ReadHeaderTimeout?

    // ReadHeaderTimeout is the amount of time allowed to read
    // request headers. The connection's read deadline is reset
    // after reading the headers and the Handler can decide what
    // is considered too slow for the body.
Copy link
Member Author

@FiloSottile FiloSottile commented Dec 15, 2016

One possible solution would be for Request.Body to be documented to allow Close concurrent with Read, and not to panic on double Close, and for ResponseWriter to be upgradeable to io.Closer.

Copy link

@peter-mogensen peter-mogensen commented Dec 16, 2016

For what it's worth ... I think we have a related use case. (only for reading requests and not for writing)

We have some slow clients doing large PUT requests - sometime on unstable connections which dies.
But we would like to allow these PUT requests as long as there is actually progress and Read() returns data. Preferably only on the endpoints where that should be allowed.

Currently, though we provide a net.Listener/net.Conn which sets Deadline on every Read/Write ... but that seems not a viable solution since it would interfere with timeouts set by net/http.Server.

Copy link

@peter-mogensen peter-mogensen commented Feb 22, 2017

Would that work? Do all your I/O in separate goroutines, and panic if they take too long?

The problem is defining what is "too long".
Often what you want is not to kill long running IO in absolute terms, but to kill "too slow" connections.
I don't want to kill a long running PUT request as long as it's actually transferring data. But I would kill a request only sending 1 byte/second.

These demands for IO activity should (for HTTP) only be enforced during ConnState stateActive (and stateNew). It would be OK for a connection to not have any IO activity in stateIdle.

I've been doing some experiments setting deadlines on connections - which is defeated by tls.Conn hiding the underlying connection object from external access.
Also trying to have a reaper go-routine Close connections with no IO activity. ... which becomes equally messy, although not impossible. (**)

Being able to set demands for IO activity during HTTP stateActive - or a more general way to do this for net.Conn without overwriting deadlines set by other API and regardless of whether the conn object is wrapped in TLS. - would be nice :)

** PS: ...much of the complexity comes from not being able to access the underlying connection in a crypto/tls.Conn object. I can understand why not exposing the connection to a Handler, but in the ConnState callback, is there any reason not to allow getting the underlying connection for a tls.Conn?

Copy link

@pwaller pwaller commented Feb 22, 2017

Just to throw in my 2¢, I also have personally hit the need to set timeouts on infinite connections, and have met someone at a go meetup who independently brought up the same problem.

In terms of API, why can't the underlying ResponseWriter implement Set{Read,Write}Deadline? It seems to me that they could happily do the write thing of sending STREAM_RST for http2. Just because it happens to implement the same API you'd use to timeout TCP connections doesn't mean it has to time out the TCP connection itself if there is another layer that can be used.

I agree with @FiloSottile's comment about the separate goroutine seeming upside-down. Though I sort of like it, it uses go's primitives in a clever non-obvious way which makes me feel nervous and seems like it would be hard to document.

Copy link

@peter-mogensen peter-mogensen commented Feb 22, 2017

Having ResponseWriter have Set*Deadline() would only solve to OP problem of writing data to the client. - not the problem of stopping long running PUT which don't really transfer data at any satisfying rate.

Copy link

@pwaller pwaller commented Feb 22, 2017

How about (*http.Request) or (*http.Request).Body implementing Set*Deadline?

Copy link

@peter-mogensen peter-mogensen commented Feb 22, 2017

If that can be implemented without interfering with other deadlines, like ReadTimeout, then it would be interesting to try.

Copy link

@gaillard gaillard commented Mar 3, 2017

@bradfitz Is it expected with the current master of the http2 pkg that when panicing out of a handler and a left behind goroutine blocked on a body read or response writer write, for a panic to occur that is internal to the http2 pkg? (existing TimeoutHandler would have same issue). Or is the behavior your referring to what would be made safe if that is the chosen route, but doesn't exist today?

Copy link

@adg adg commented Mar 10, 2017

We ran into this issue on our project: upspin/upspin#313

We had our timeouts set quite low, as per @FiloSottile's recommendations but our users regularly send 1MB POST requests, which not all connections can deliver in a short time.

We would like to be able to set the timeout after reading the request headers. This would let us increase the timeouts for authenticated users only (we don't mind if our own users want to DoS us; at least we'll know who they are), but give unauthenticated users a very short timeout.

cc @robpike

Copy link
Member Author

@FiloSottile FiloSottile commented Jun 11, 2018

So if I understand correctly the plan is to set a timeout on the main goroutine with a panic at the end, and do your I/O in a new goroutine.

I see a few issues with this:

  1. It doesn't (currently) work. If you run the Play link above the I/O goroutine remains blocked on the slow reader after the handler panics. http.ResponseWriter.Write should unblock and return errors once ServeHTTP returns. This is arguably just a bug.
  2. It's convoluted and hard to discover for something (timeouts) that we would want most Handlers to manage well. It feels like net/http internals spilled over.
  3. It feels very easy to get wrong, losing track of a goroutine, or something like that.
  4. It requires special behavior from the http.ResponseWriter provider (see 1) that was not in the interface before.
Copy link

@bradfitz bradfitz commented Jun 11, 2018

@FiloSottile, as background, we're asking whether this is enough for now because there's no great place to put new options. We want to stop adding optional interfaces for http.ResponseWriter. They compose poorly.

If we make a "v2" version of net/http, we can fix this (and many other things), but we're wondering if we really need anything more for now.

If you feel strongly we do need this now, then what API do you propose?

Copy link

@bradfitz bradfitz commented Jun 12, 2018

I was about to propose a hypothetical API but I realized I wasn't sure whether I had the right bug so I just tried to reread this whole bug to find the problem statement and I'm mostly seeing solutions.

Can somebody summarize the problem we're trying to solve here?

Copy link
Member Author

@FiloSottile FiloSottile commented Jun 12, 2018

The shortest tl;dr is that there is no way to set timeouts based on the endpoint or on the client, but only server-wide, nor there is any way to update timeouts while serving a response based on the changing circumstances.

The reason one wants to set timeouts is to protect against clients going away or being intentionally slow. A reason one might want to exempt an endpoint is for example if that handler is serving streaming content or long downloads. A reason one might want to exempt a client is for example following authentication.

#16100 (comment) is a good practical example. I feel like the first two paragraphs of my original report are also not solution-oriented.

Copy link

@bradfitz bradfitz commented Jun 12, 2018

Okay, thanks.

If the client goes away ("laptop lid closes in a coffee shop" case), our default TCP keep-alive detection will kill the connection after some time. So I'm less worried about that.

But having different policies based on authentication is a good example.

I imagine this would be equally about read timeouts in addition to write timeouts.

How about something like this for an API: (ignoring the names)

package http // net/http

func NewHandlerController(w http.ResponseWriter, r *http.Request) *HandlerController { ... }

type HandlerController struct { /* opaque */ }

// SetFoo changes the foo deadline and reports whether it was successfully adjusted.
func (hc *HandlerController) SetFoo(time.Time) bool { ... }
func (hc *HandlerController) SetBar(time.Duration) bool { ... }

And then we're largely insulated from the mechanism we use to implement it over time. If we want to do private magic on the ResposneWriter interface, we can. If we want to do magic private context values, we can. But the user gets concrete types.

I'd start with that (for Go 1.12) and implement it for HTTP/1 and HTTP/2 only and not provide a way for others to implement it for their own Servers/ResponseWriters right away, knowing that this API would let us do so later.

I'm not really a fan of the name HandlerController name, though, but it's verbose here as an example.

Copy link
Member Author

@FiloSottile FiloSottile commented Jun 12, 2018

LGTM, although I would return an error (Set[Read|Write]Deadline(time.Time) error) so that we can communicate why the feature is not available.

Copy link

@edaniels edaniels commented Jul 15, 2018

Pulling together a few comments over time here. Given the ResponseWriter is what currently let's us hijack the underlying connection, would it be sufficient to start having net/http's ResponseWriterimplementio.Closerand a newhttp.DeadlineController` with methods:

type DeadlineController interface {

This would be in line with the other interfaces already in use and implemented by the underlying ResponseWriter from the server.

For a user wanting to implement Server-Side Events, they could do the following:

func streamHandler(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "text/event-stream")
	controller, ok := w.(http.DeadlineController)
	if !ok {
		panic("expected w to be a http.DeadlineController")
	for  {
		w.Write([]byte("data: some piece of data that takes up to 2 seconds to write\n\n"))

You could mix in io.Closer into that example or use the request.Context to signal the handler to stop the stream.

I do also like @tombergan's mention of contextified reads and writes as potentially new methods so as to avoid breaking the library.

With all that said, it feels like the right future approach is to keep functionality scoped to the ResponseWriter and not at a higher level (handler, router, or server) for the motivation of having the most flexibility and obvious/expected API.

Copy link

@pd60193 pd60193 commented Jul 27, 2018

The shortest tl;dr is that there is no way to set timeouts based on the endpoint or on the client, but only server-wide, nor there is any way to update timeouts while serving a response based on the changing circumstances.

I want to do this. I have an endpoint which exports a large file while others perform operations on smaller sized data. I would like to keep different timeouts for these two different endpoints.

Copy link

@cespare cespare commented Nov 2, 2018

So it it okay for anyone send a CL for the API @bradfitz described in #16100 (comment)? (And we'll bikeshed the naming there?)

Copy link

@marco-m marco-m commented Feb 18, 2019

Hello any news ?

Copy link

@LasTshaMAN LasTshaMAN commented Apr 23, 2019

Any updates on this ?

Copy link

@fn-code fn-code commented Jun 20, 2019

Hi Any updates ??

Copy link

@mikelnrd mikelnrd commented Aug 1, 2019

@FiloSottile (author of this issue) wrote a great blogpost on cloudflare's blog about this. It's not linked anywhere here so I thought I'd include it for any interested readers.

My interest in this is being able to continuously update the timeout (to a few seconds in the future each time) on SSE server-sent-event streaming handlers while keeping deadlines for other regular non-streaming handlers much shorter. Hope to see some movement on this soon. EDIT: exactly like this: #16100 (comment)

Copy link

@rep-movsd rep-movsd commented Aug 15, 2019

Any update at all?

Copy link

@cumirror cumirror commented Aug 19, 2019

maybe we could use the callback in http server structure:

	// ConnState specifies an optional callback function that is
	// called when a client connection changes state. See the
	// ConnState type and associated constants for details.
	ConnState func(net.Conn, ConnState)

states change like this:
after conn accept -> StateNew
read some bytes -> StateActive // loop on the same connection
after handle -> StateIdle

when the first request come in, we just set a default timeout through ConnState.
After we have handled the request in ServeHTTP, we have known the timeout through our rule, such as path or host, then record a map such as map[remoteAddr]timeout.
Then when other requestes come on the same connection, we could set the corrected timeout through ConnState.

we could not know the timeout until we have handled the request in ServeHTTP, so the first arrival request will use default timeout, other requests on the same connection will set correctly.

moreover, we could not set SetWriteDeadline(time.Now().Add(d)) just after read, so write timeout is inaccuracy. And this operation also conflict with server's WriteTimeout and readTimeout unless we don't set these timeout.

Hmmm... just forgot ConnState, it is not a good way.

Copy link

@mikelnrd mikelnrd commented Sep 29, 2019

Now that Server.ConnContext has been added in go 1.13, the solution of the issue's original author @FiloSottile to expose the net.Conn in the Context is I think more viable:

// ConnContext optionally specifies a function that modifies
// the context used for a new connection c. The provided ctx
// is derived from the base context and has a ServerContextKey
// value.
ConnContext func(ctx context.Context, c net.Conn) context.Context // Go 1.13

Approach: Create a *DeadlineController inside of Server.ConnContext and embed it into the Context.

func NewDeadlineController(c net.Conn, s *http.Server) *DeadlineController {
	return &DeadlineController{
		c: c,
		s: s,
type DeadlineController struct {
	c net.Conn
	s *http.Server

// ExtendWriteDeadline can be called to reset the write deadine.
// The deadline will be extended as if a new request had come in.
// Can be called repeatedly to extend the deadline forever.
func (dc *DeadlineController) ExtendWriteDeadline() error {
	return dc.c.SetWriteDeadline(time.Now().Add(dc.s.WriteTimeout))

I'm thinking of using the above code to extend the write deadline (possibly indefinitely) inside of a streaming http.ResponseWriter (which is sending Server-Sent-Events to a browser in my case). The goal is to be able to have short deadlines on other routes for security purposes (see cloudflare's blogposts linked in this issue for more details on that).

You could do something similar for the read deadline.

Request for any comments: Are there any issues with this approach? Will it work for http1 and http2? Is there a better solution we can work towards?

Copy link

@aead aead commented Jan 23, 2020

I (again) run into the issue of specifying handler-specific timeouts.

@mikelnrd AFAICS your approach does not work since the connection you get from ConnContext func(ctx context.Context, c net.Conn) context.Context is the TCP / TLS connection. However, the write timeout closes the (private) HTTP connection. At least I was not able to get this working - either all connections stick around forever or get closed after the WriteTimeout...

However, I may have found a similar work-a-round:

ConnContext: func(ctx context.Context, c net.Conn) context.Context {
	writeTimeout, cancelWriteTimeout := context.WithTimeout(ctx, 10*time.Second)
	go func() {
		defer cancelWriteTimeout() // Release resources
		_ = <-writeTimeout.Done() // Wait for the timeout or the timeout cancel
		if err := writeTimeout.Err(); err == context.DeadlineExceeded {
		     c.Close() // Only close the connection in case of exceeded deadline - not in case of cancellation
	return context.WithValue(ctx, ctx.Value(http.ServerContextKey), cancelWriteTimeout)

Here, I create a go routine that waits for the writeTimeout. If the deadline exceeds, we close the connection since that's the purpose of the write timeout.
However, if the timeout is canceled then we don't close the connection. This allows handler functions which want to opt-out of write timeouts call the cancelWriteTimeout function - which they can access via http.Request.Context()

At the moment this approach has the problem that if a handler function returns (e.g. the request has been handled successfully) the write-timeout go routine will stick around until the timeout elapses.
(Unfortunately, we can't use a select with <-ctx.Done() here - See: ListenAndServe implementation)

Therefore, I wrap the server.Handler with:

type cancelWriteTimeoutHandler struct{ *http.ServeMux }

func (h cancelWriteTimeoutHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	defer func() {
		ctx := r.Context()
		f := ctx.Value(ctx.Value(http.ServerContextKey))
		if f, ok := f.(context.CancelFunc); ok {
	h.ServeMux.ServeHTTP(w, r)

This handler will cancel the write timeout when the request has been handled. This ensures that the write-timeout go routine only lives as long as the request.
For a minimal example see:
A cleaner solution would be nice but this seems to work 🤞

Copy link

@perillo perillo commented Feb 21, 2020

nginx have the send_timeout option:

Sets a timeout for transmitting a response to the client. The timeout is set only between two successive write operations, not for the transmission of the whole response. If the client does not receive anything within this time, the connection is closed.

This can be implemented with a new Server.SendTimeout field.

Copy link

@tv42 tv42 commented Feb 21, 2020

@perillo Settings for the whole server are explicitly not what this issue asks for. There's a need for handler-specific timeouts.

(Secondly, the idea of a write-per-timeout is still open to slowloris abuse)

Copy link

@lauri-elevant lauri-elevant commented Jun 19, 2020

This issue needs attention.

There are many reasonable use cases, like different policies depending on authentication or varied types of requests. In my case, a single service is supposed to usually serve small responses, but sometimes after authentication and understandind the request, it turns out that the file to be served is huge and might take some legitimate user hours to download.

So you have to balance between:

  1. Setting a "high enough" write timeout for most users to succeed
  2. Setting a "low enough" write timeout to not be totally exposed to a slow lori attack
    So you end up with an intersection of two very bad choices.

It is hard to comprehend how there is no way to keep a legitimate connection going, while giving unauthenticated users a proper timeout. I cannot understand how people are able to overlook this and expose Go services to the internet relying on net/http.

Is there any acceptable workaround besides having a reverse proxy babysitting the Go service? Or another package to drop in place of net/http?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.