Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: timeouts for file uploads #39468

Closed
ItalyPaleAle opened this issue Jun 9, 2020 · 11 comments
Closed

net/http: timeouts for file uploads #39468

ItalyPaleAle opened this issue Jun 9, 2020 · 11 comments

Comments

@ItalyPaleAle
Copy link

@ItalyPaleAle ItalyPaleAle commented Jun 9, 2020

Using go 1.14

In the net/http package, there seems not to be a reliable way to put a timeout for requests to identify stalled ones when the user is uploading a relatively large file. This is an issue on both the server and client that use HTTP requests using net/http.

On the server side, every possible timeout that can be set could cause requests to be interrupted if the user is uploading a large-enough file.

On the client side, the same thing happens: if users are uploading a large file, they might trigger a timeout. Every possible timeout that can be set using the net/http package could halt requests.

Of course, I could set a "very long" timeout. However, the definition of "very long" is subjective, as users who are on a slow connection could take hours to upload files of a certain size.

Ideally, there should be a timeout that identifies connections that are taking too long without considering the time clients are using to transmit data to the server.

@sebnyberg
Copy link

@sebnyberg sebnyberg commented Jun 11, 2020

Why not use a context with a timeout?

ctx, cancel := context.WithTimeout(ctx, time.Hour * 2)
defer cancel()

// This request returns an error after 2 hours
req, err := http.NewRequestWithContext(ctx, "GET", "https://some.url.com", nil)
if err != nil {
	log.Fatal(err)
}

resp, err := http.DefaultClient.Do(req)

If you have a more complex scenario where the timeout is not known, perform the request asynchronously and run some logic on given interval to see whether the request should be canceled:

func MakeRequest(ctx context.Context) (*http.Response, error) {
	ctx, cancel := context.WithCancel(ctx)
	defer cancel()

	req, err := http.NewRequestWithContext(ctx, "GET", "https://some.url.com", nil)
	if err != nil {
		return nil, err
	}
	type Result struct {
		resp *http.Response
		err  error
	}
	resultch := make(chan Result)
	go func() {
		defer close(resultch)
		resp, err := http.DefaultClient.Do(req)
		resultch <- Result{resp, err}
	}()

	// Returning from within this loop triggers defer cancel(),
	// effectively shutting down downstream consumers of the context
	for {
		select {
		// Return when request finishes
		case result := <-resultch:
			return result.resp, result.err
		// Check every minute if request should be canceled using some logic
		case <-time.After(time.Minute):
			if checkConnectionIsStale(someArgs) {
				return nil, errors.New("cancelling connection, was stale")
			}
		// Stop if the parent has closed the context
		case <-ctx.Done():
			return nil, errors.New("request canceled by parent")
		}
	}
}
@ItalyPaleAle
Copy link
Author

@ItalyPaleAle ItalyPaleAle commented Jun 11, 2020

That is a very interesting approach to set a timeout on the client. And thanks for sharing the code!

Not sure how easy it would be to implement it on the server, however?

In general, it would be nice if this was part of the standard library

@sebnyberg
Copy link

@sebnyberg sebnyberg commented Jun 11, 2020

Not sure how easy it would be to implement it on the server, however?

The example I wrote above is honestly the first time I've used a context on the client-side. It is used everywhere in Go to manage timeouts, deadlines and cancellation in a concurrency-safe way. Most guides you'll find will urge you to initialize a context with each request in the backend. Basically it's like throwing a rope with a timeout downstream and being able to pull it up again, canceling all child actions along the way, whether it be a HTTP request, a database query, or some other long-running task doesn't really matter.

@ItalyPaleAle
Copy link
Author

@ItalyPaleAle ItalyPaleAle commented Jun 12, 2020

My request was for implementing in the standard library a way to handle timeouts (both in clients and servers) for HTTP requests which have large bodies, so the timeout wouldn’t hit if the client is just slow (eg 3G users)

@toothrot
Copy link
Contributor

@toothrot toothrot commented Jun 12, 2020

We mainly use our issue tracker for tracking bugs or proposals. I recommend asking your question on one of the forums listed here: http://golang.org/wiki/Questions, where it will reach a broader audience. I am sure someone in one of those forums can help you configure fine-grained timeouts around reading request bodies in your server, or writing request bodies in your client.

@toothrot toothrot closed this Jun 12, 2020
@ItalyPaleAle
Copy link
Author

@ItalyPaleAle ItalyPaleAle commented Jun 12, 2020

Hi @toothrot thanks for responding. This issue is a proposal/feature request:

My request was for implementing in the standard library a way to handle timeouts (both in clients and servers) for HTTP requests which have large bodies, so the timeout wouldn’t hit if the client is just slow (eg 3G users)

I have searched throughout the net/http package and couldn’t find anything that would do this reliably.

Every timeout available in net/http is purely time-based and does not consider the case in which a client is transferring a lot of data to the server (which ideally shouldn’t be part of the timeout, if the client is still transferring data: they might just be slow).

@networkimprov
Copy link

@networkimprov networkimprov commented Jun 12, 2020

Maybe a minimum bits/sec threshold would be helpful?

@toothrot, pls reopen, thx!

@toothrot
Copy link
Contributor

@toothrot toothrot commented Jun 12, 2020

The reason why I closed this is that reading and writing of bytes happens in the application code reading from Response.Body, or writing to Request.Body, not in the net/http library itself. Any timeout, or throughput based control would likely best be handled in application code in my experience.

/cc @bradfitz as an owner.

@toothrot
Copy link
Contributor

@toothrot toothrot commented Jun 12, 2020

I also recommend following the discussion in #16100 for a longer discussion about why this works the way it is, and proposals to change it.

@ItalyPaleAle
Copy link
Author

@ItalyPaleAle ItalyPaleAle commented Jun 12, 2020

@toothrot I see #16100 and I think that is trying to solve the same problem. Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.