Skip to content

net/http: Docs update for connection reuse #26095

Closed
@firefart

Description

@firefart

What version of Go are you using (go version)?

go version go1.10.2 windows/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\firefart\AppData\Local\go-build
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\firefart\go
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\firefart\AppData\Local\Temp\go-build959778686=/tmp/go-build -gno-record-gcc-switches

Suggested Change

Currently the docs on https://golang.org/pkg/net/http/ say that you need to close the response Body after you are done with it so golang can reuse the connection. In my tests it turned out, that you have to actually consume the body before closing it otherwise the connection will not be reused.
I think it should be stated everywhere in the referenced docs where Close is mentioned that you have to consume the body too. If you do not consume the body golang will create new connections on every request.
If you aren't interested in the body, you need to do smth like

io.Copy(ioutil.Discard, resp.Body)
defer resp.Body.Close()

This blog post also describes the problem:
https://awmanoj.github.io/tech/2016/12/16/keep-alive-http-requests-in-golang/

Here is a sample program to demonstrate the problem:

package main

import (
	"flag"
	"io"
	"io/ioutil"
	"net/http"
	"sync"
	"time"
)

var (
	client = &http.Client{
		Timeout: 30 * time.Second,
	}
	discard bool
	threads int
	url     string
	wg      sync.WaitGroup
	c       chan struct{}
)

func main() {
	flag.BoolVar(&discard, "discard", false, "Consume body")
	flag.IntVar(&threads, "threads", 8, "threads")
	flag.StringVar(&url, "url", "", "url")
	flag.Parse()
	wg.Add(threads)
	c = make(chan struct{}, threads)
	for i := 0; i < threads; i++ {
		go func() {
			for range c {
				req, _ := http.NewRequest(http.MethodGet, url, nil)
				resp, err := client.Do(req)
				if err == nil {
					if discard {
						io.Copy(ioutil.Discard, resp.Body)
					}
					resp.Body.Close()
				}
			}
			wg.Done()
		}()
	}

	for i := 0; i < 200; i++ {
		c <- struct{}{}
	}
	close(c)
	wg.Wait()
}

Here is the timing output from only closing the body, and the other time with consuming it before. Each test was executed twice to make sure it's no network based problem. Also a https URL was used to demonstrate the problem because the TLS handshake takes a lot of time.

PS C:\Users\firefart\go\src\> Measure-Command { .\test.exe -url https://firefart.at }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 3
Milliseconds      : 487
Ticks             : 34877955
TotalDays         : 4,03680034722222E-05
TotalHours        : 0,000968832083333333
TotalMinutes      : 0,058129925
TotalSeconds      : 3,4877955
TotalMilliseconds : 3487,7955



PS C:\Users\firefart\go\src\> Measure-Command { .\test.exe -url https://firefart.at }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 3
Milliseconds      : 511
Ticks             : 35116646
TotalDays         : 4,06442662037037E-05
TotalHours        : 0,000975462388888889
TotalMinutes      : 0,0585277433333333
TotalSeconds      : 3,5116646
TotalMilliseconds : 3511,6646



PS C:\Users\firefart\go\src\> Measure-Command { .\test.exe -url https://firefart.at -discard }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 948
Ticks             : 9482683
TotalDays         : 1,09753275462963E-05
TotalHours        : 0,000263407861111111
TotalMinutes      : 0,0158044716666667
TotalSeconds      : 0,9482683
TotalMilliseconds : 948,2683



PS C:\Users\firefart\go\src\> Measure-Command { .\test.exe -url https://firefart.at -discard }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 904
Ticks             : 9042128
TotalDays         : 1,04654259259259E-05
TotalHours        : 0,000251170222222222
TotalMinutes      : 0,0150702133333333
TotalSeconds      : 0,9042128
TotalMilliseconds : 904,2128

You can also verify this behaviour using wireshark and have a look at the source port. If it changes on every request there is no connection reuse.

So it would be great if you can add this case to the net/http docs

Metadata

Metadata

Assignees

No one assigned

    Labels

    DocumentationIssues describing a change to documentation.FrozenDueToAgeNeedsFixThe path to resolution is known, but the work has not been done.SuggestedIssues that may be good for new contributors looking for work to do.

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions