Skip to content

net/http: Transport.MaxConnsPerHost doesn't work well with HTTP/2 #27753

Closed
@gaboose

Description

@gaboose

What version of Go are you using (go version)?

go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOHOSTARCH="amd64"
GOHOSTOS="linux"

What did you do?

I ran a program that quickly and concurrently executes 10 http requests to the same url from a single http.Client, also httptraceing TLSHandshakeStart and GotConn events.

package main

import (
	"fmt"
	"net/http"
	"net/http/httptrace"
	"sync"
)

func checkError(err error) {
	if err != nil {
		panic(err)
	}
}

func main() {
	client := &http.Client{}
	req, err := http.NewRequest("GET", "https://golang.org/doc/", nil)
	checkError(err)

	trace := &httptrace.ClientTrace{
		GotConn: func(connInfo httptrace.GotConnInfo) {
			fmt.Printf("Got Conn: %+v\n", connInfo)
		},
		TLSHandshakeStart: func() {
			fmt.Println("TLSHandshakeStart")
		},
	}
	req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))

	wg := sync.WaitGroup{}
	for i := 0; i < 10; i++ {
		wg.Add(1)
		go func() {
			_, err := client.Do(req)
			checkError(err)
			wg.Done()
		}()
	}
	wg.Wait()
}

What did you expect to see?

TLSHandshakeStart being printed as many times as GotConn: {[...] Reused:false [...]} - only once.

What did you see instead?

TLSHandhshakeStart is printed 10 times, but GotConn reports that it reused the connection 9 times out of 10.

This is the full output:

TLSHandshakeStart
TLSHandshakeStart
TLSHandshakeStart
TLSHandshakeStart
TLSHandshakeStart
TLSHandshakeStart
TLSHandshakeStart
TLSHandshakeStart
TLSHandshakeStart
TLSHandshakeStart
Got Conn: {Conn:0xc4202b6000 Reused:false WasIdle:false IdleTime:0s}
Got Conn: {Conn:0xc4202b6000 Reused:true WasIdle:false IdleTime:0s}
Got Conn: {Conn:0xc4202b6000 Reused:true WasIdle:false IdleTime:0s}
Got Conn: {Conn:0xc4202b6000 Reused:true WasIdle:false IdleTime:0s}
Got Conn: {Conn:0xc4202b6000 Reused:true WasIdle:false IdleTime:0s}
Got Conn: {Conn:0xc4202b6000 Reused:true WasIdle:false IdleTime:0s}
Got Conn: {Conn:0xc4202b6000 Reused:true WasIdle:false IdleTime:0s}
Got Conn: {Conn:0xc4202b6000 Reused:true WasIdle:false IdleTime:0s}
Got Conn: {Conn:0xc4202b6000 Reused:true WasIdle:false IdleTime:0s}
Got Conn: {Conn:0xc4202b6000 Reused:true WasIdle:false IdleTime:0s}

The large number of TLS handshakes is a big performance hit for us. This doesn't happen if I specify an http2 transport explicitly:

- client := &http.Client{}
+ client := &http.Client{Transport: &http2.Transport{}}

Additionally strace -f -e trace=network -s 10000 ./mytestprogram prints ~10 times less connect syscalls if I specify the http2 transport explicitly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    FrozenDueToAgeNeedsFixThe path to resolution is known, but the work has not been done.

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions