Description
What version of Go are you using (go version
)?
go version go1.10.2 windows/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env
)?
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\firefart\AppData\Local\go-build
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\firefart\go
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\firefart\AppData\Local\Temp\go-build959778686=/tmp/go-build -gno-record-gcc-switches
Suggested Change
Currently the docs on https://golang.org/pkg/net/http/ say that you need to close the response Body after you are done with it so golang can reuse the connection. In my tests it turned out, that you have to actually consume the body before closing it otherwise the connection will not be reused.
I think it should be stated everywhere in the referenced docs where Close is mentioned that you have to consume the body too. If you do not consume the body golang will create new connections on every request.
If you aren't interested in the body, you need to do smth like
io.Copy(ioutil.Discard, resp.Body)
defer resp.Body.Close()
This blog post also describes the problem:
https://awmanoj.github.io/tech/2016/12/16/keep-alive-http-requests-in-golang/
Here is a sample program to demonstrate the problem:
package main
import (
"flag"
"io"
"io/ioutil"
"net/http"
"sync"
"time"
)
var (
client = &http.Client{
Timeout: 30 * time.Second,
}
discard bool
threads int
url string
wg sync.WaitGroup
c chan struct{}
)
func main() {
flag.BoolVar(&discard, "discard", false, "Consume body")
flag.IntVar(&threads, "threads", 8, "threads")
flag.StringVar(&url, "url", "", "url")
flag.Parse()
wg.Add(threads)
c = make(chan struct{}, threads)
for i := 0; i < threads; i++ {
go func() {
for range c {
req, _ := http.NewRequest(http.MethodGet, url, nil)
resp, err := client.Do(req)
if err == nil {
if discard {
io.Copy(ioutil.Discard, resp.Body)
}
resp.Body.Close()
}
}
wg.Done()
}()
}
for i := 0; i < 200; i++ {
c <- struct{}{}
}
close(c)
wg.Wait()
}
Here is the timing output from only closing the body, and the other time with consuming it before. Each test was executed twice to make sure it's no network based problem. Also a https URL was used to demonstrate the problem because the TLS handshake takes a lot of time.
PS C:\Users\firefart\go\src\> Measure-Command { .\test.exe -url https://firefart.at }
Days : 0
Hours : 0
Minutes : 0
Seconds : 3
Milliseconds : 487
Ticks : 34877955
TotalDays : 4,03680034722222E-05
TotalHours : 0,000968832083333333
TotalMinutes : 0,058129925
TotalSeconds : 3,4877955
TotalMilliseconds : 3487,7955
PS C:\Users\firefart\go\src\> Measure-Command { .\test.exe -url https://firefart.at }
Days : 0
Hours : 0
Minutes : 0
Seconds : 3
Milliseconds : 511
Ticks : 35116646
TotalDays : 4,06442662037037E-05
TotalHours : 0,000975462388888889
TotalMinutes : 0,0585277433333333
TotalSeconds : 3,5116646
TotalMilliseconds : 3511,6646
PS C:\Users\firefart\go\src\> Measure-Command { .\test.exe -url https://firefart.at -discard }
Days : 0
Hours : 0
Minutes : 0
Seconds : 0
Milliseconds : 948
Ticks : 9482683
TotalDays : 1,09753275462963E-05
TotalHours : 0,000263407861111111
TotalMinutes : 0,0158044716666667
TotalSeconds : 0,9482683
TotalMilliseconds : 948,2683
PS C:\Users\firefart\go\src\> Measure-Command { .\test.exe -url https://firefart.at -discard }
Days : 0
Hours : 0
Minutes : 0
Seconds : 0
Milliseconds : 904
Ticks : 9042128
TotalDays : 1,04654259259259E-05
TotalHours : 0,000251170222222222
TotalMinutes : 0,0150702133333333
TotalSeconds : 0,9042128
TotalMilliseconds : 904,2128
You can also verify this behaviour using wireshark and have a look at the source port. If it changes on every request there is no connection reuse.
So it would be great if you can add this case to the net/http docs