net/http: don't block RoundTrip when the Transport hits MaxConcurrentStreams

[as]

The CL that disables connection pooling for HTTP2 creates a significant discontinuity in throughput when the server specifies a small number of maximum concurrent streams.

https://go-review.googlesource.com/c/net/+/53250

HTTP2 support is automatically enabled in Go under conditions not always specified by the developer. For example, configuration files often alternate between http and https endpoints. When using an http endpoint, go will use HTTP/1, whereas https endpoints use HTTP/2.

The HTTP/1 default transport will create as many connections as needed in the background. The HTTP2 default transport does not (although it used to).

As a result, HTTP1 endpoints get artificially high throughput when compared to HTTP2 endpoints that block waiting for more streams to become available instead of creating a new connection. For example, the AWS ALB limit the maximum number of streams to 128.

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-listeners.html

This HTTP/2 client is blocked once it hits 128 streams and waits for more to become available. The HTTP/1 client does not. The performance of the HTTP/1 client is orders of magnitude faster as a result. This effect is annoying and creates a leaky abstraction in the net/http package.

The consequence of this is that importers of the net/http package now have to:

1.) Distinguish between HTTP and HTTPS endpoints
2.) Write a custom connection pool for the transport when HTTP2 is enabled

I think the previous pooling functionality should be restored.

[dgryski]

/cc @bradfitz @tombergan

[meirf]

If we revert to the old behavior, one lowish priority thing we can do is have MaxConnsPerHost apply to the total number of h2 connections as opposed to currently only applying to h2 dials. That way, there can be an opted-in limit.

[theckman]

There looks to be a bit of overlap with #17776. Just wanted to make sure the two issues get linked together.

[twmb]

FWIW, this is one of the reasons I disable http2 with TLSNextProto in a project.

[itsjamie]

If a user has it configured to allow an infinite number of connections per host, then I believe the H2 implementation then should create a new connection when it has reached the server-advertised maximum.

However, there is some level of overhead in creating a new connection, perhaps a deadline on the RoundTrip block to attempt to stay in the server configured maxStreams before offloading the request to a new connection?

[smira]

We're facing problem with this change as well, as we're sending HTTP requests and responses with huge bodies, and performance isn't optimal as all of them are serialized over a single TCP connection. We expect to have around 10 clients against single server endpoint, so with current approach it means 10 concurrent TCP connections overall (1 connection per client-server pair). Limiting MaxConcurrentStreams on server side leads to client requests being blocked while waiting for connection.

Right now in test environment, client is a load-test with N goroutines sending HTTP requests concurrently. These all requests are serialized and queued over a single connection, which leads to actual bandwidth drop when concurrency is increased at some point.

[smira]

To add some numbers, with unpatched HTTP/2 library we get around 1.5Gbit/s (up+down), with same settings and parts of the patch removed to allow more connections to be open we get 7Gbit/s (up+down) over several connections.

Number of concurrent streams is limited on server side to 30, this might not be the optimal number, we'll keep testing but definitely +1 for this change to be reverted or making it configurable.

[liggitt]

it seems like the client is assuming that all future connections to a given host will hit the same backend that told the client how many concurrent streams it could send over the first connection... that assumption doesn't hold for load balanced services where the same host can be serviced by multiple backends.

A limit for max connections per host makes sense, but MaxConcurrentStreams isn't a great stand-in for it.

[twmb]

Nothing prevents a load balancer from replying with a high max concurrent streams. The problem I see is h2 implementations using the recommended minimum SETTINGS_MAX_CONCURRENT_STREAMS for no reason. I would guess this is because implementors simply choose the only value mentioned. I've also not seen yet an implementation that re-negotiates this value upwards.

Go's current h2 bundle uses a value of 1000 for client connections by default, but only 250 for server connections. Further, the server stream comment refers to the Google Front End using a default value of 100—and I would think that Google servers can handle more than 100 concurrent requests, regardless of whether they are from one host, or, say, a proxy server.

[Gobd]

We're seeing significantly reduced throughput after this change using http://google.golang.org/api/bigquery/v2 to stream data into BigQuery.

[rsc]

Can someone say what the HTTP/2 spec says about this setting? (Brad says it doesn't say.)
Failing that, can someone say what the big browsers do with this setting?