Skip to content

net/http: the client only use one connection while downloading from the server concurrency #40944

Closed
@ComfyFluffy

Description

@ComfyFluffy

What version of Go are you using (go version)?

$ go version
go version go1.15 windows/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\user\AppData\Local\go-build
set GOENV=C:\Users\user\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\user\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\user\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\user\AppData\Local\Temp\go-build545182214=/tmp/go-build -gno-record-gcc-switches

What did you do?

I am downloading images from i.pximg.net with multiple goroutines.
My code:

package main

import (
	"fmt"
	"io"
	"io/ioutil"
	"log"
	"net/http"
	"sync"
)

func main() {
	wg := sync.WaitGroup{}
	client := &http.Client{Transport: &http.Transport{}}
	for i := 0; i < 5; i++ {
		wg.Add(1)
		go func(i int) {
			defer wg.Done()
			log.Println(i, "started")
			url := fmt.Sprintf("https://i.pximg.net/img-original/img/2019/09/13/00/14/39/76752268_p%d.jpg", i)
			// url := "http://127.0.0.1/test-http"
			req, err := http.NewRequest("GET", url, nil)
			// Set the Referer header so we will not get 403 error
			req.Header["Referer"] = []string{"https://www.pixiv.net/"}
			res, err := client.Do(req)
			if err != nil {
				log.Println(err)
				return
			}
			defer res.Body.Close()
			log.Println(i, "downloading", res.Header, res.ContentLength, res.Status)
			wr, err := io.Copy(ioutil.Discard, res.Body)
			log.Println(i, wr, err)
		}(i)
	}
	wg.Wait()
}

What did you expect to see?

The client will create 5 connections to the server and download the content.

What did you see instead?

The client created 5 connections to the server at first, but 4 connections closed immediately. Only one connection was used to transfer the data.

Output:

2020/08/21 10:46:25 4 started
2020/08/21 10:46:25 3 started
2020/08/21 10:46:25 1 started
2020/08/21 10:46:25 2 started
2020/08/21 10:46:25 0 started
2020/08/21 10:46:27 1 downloading map[Accept-Ranges:[bytes] Age:[136306] Cache-Control:[max-age=31536000] Content-Length:[618736] Content-Type:[image/jpeg] Date:[Fri, 21 Aug 2020 02:46:28 GMT] Expires:[Thu, 19 Aug 2021 12:54:42 GMT] Last-Modified:[Thu, 12 Sep 2019 15:14:39 GMT] Server:[nginx] Via:[http/1.1 f002 (second)] X-Content-Type-Options:[nosniff]] 618736 200 OK
2020/08/21 10:46:28 4 downloading map[Accept-Ranges:[bytes] Age:[136248] Cache-Control:[max-age=31536000] Content-Length:[651153] Content-Type:[image/jpeg] Date:[Fri, 21 Aug 2020 02:46:28 GMT] Expires:[Thu, 19 Aug 2021 12:55:40 GMT] Last-Modified:[Thu, 12 Sep 2019 15:14:39 GMT] Server:[nginx] Via:[http/1.1 f001 (second)] X-Content-Type-Options:[nosniff]] 651153 200 OK
2020/08/21 10:46:28 2 downloading map[Accept-Ranges:[bytes] Age:[163397] Cache-Control:[max-age=31536000] Content-Length:[683337] Content-Type:[image/jpeg] Date:[Fri, 21 Aug 2020 02:46:28 GMT] Expires:[Thu, 19 Aug 2021 05:23:11 GMT] Last-Modified:[Thu, 12 Sep 2019 15:14:39 GMT] Server:[nginx] Via:[http/1.1 f001 (second)] X-Content-Type-Options:[nosniff]] 683337 200 OK
2020/08/21 10:46:28 3 downloading map[Accept-Ranges:[bytes] Age:[1255] Cache-Control:[max-age=31536000] Content-Length:[594356] Content-Type:[image/jpeg] Date:[Fri, 21 Aug 2020 02:46:28 GMT] Expires:[Sat, 21 Aug 2021 02:25:33 GMT] Last-Modified:[Thu, 12 Sep 2019 15:14:39 GMT] Server:[nginx] 
Via:[http/1.1 f005 (second)] X-Content-Type-Options:[nosniff]] 594356 200 OK
2020/08/21 10:46:28 0 downloading map[Accept-Ranges:[bytes] Age:[175766] Cache-Control:[max-age=31536000] Content-Length:[5681832] Content-Type:[image/jpeg] Date:[Fri, 21 Aug 2020 02:46:28 GMT] Expires:[Thu, 19 Aug 2021 01:57:02 GMT] Last-Modified:[Thu, 12 Sep 2019 15:14:39 GMT] Server:[nginx] Via:[http/1.1 f003 (second)] X-Content-Type-Options:[nosniff]] 5681832 200 OK
2020/08/21 10:46:29 1 618736 <nil>
2020/08/21 10:46:29 4 651153 <nil>
2020/08/21 10:46:29 2 683337 <nil>
2020/08/21 10:46:29 3 594356 <nil>
2020/08/21 10:46:31 0 5681832 <nil>

Proxifier log (An app which can force other apps to use a proxy)

[08.21 10:25:31] main.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:25:31] main.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:25:31] main.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:25:31] main.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:25:31] main.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:25:32] main.exe *64 - i.pximg.net:443 close, 438 bytes sent, 4860 bytes (4.74 KB) received, lifetime 00:01
[08.21 10:25:32] main.exe *64 - i.pximg.net:443 close, 438 bytes sent, 4860 bytes (4.74 KB) received, lifetime 00:01
[08.21 10:25:32] main.exe *64 - i.pximg.net:443 close, 438 bytes sent, 4860 bytes (4.74 KB) received, lifetime 00:01
[08.21 10:25:32] main.exe *64 - i.pximg.net:443 close, 438 bytes sent, 4860 bytes (4.74 KB) received, lifetime 00:01
[08.21 10:25:43] main.exe *64 - i.pximg.net:443 close, 42961 bytes (41.9 KB) sent, 8263702 bytes (7.88 MB) received, lifetime 00:12

I tried to build a local server with echo to test:

...
	e.GET("/test-http", func(c echo.Context) error {
		f, err := os.Open(`test.png`)
		if err != nil {
			return err
		}
		defer f.Close()
		b := ratelimit.NewBucketWithRate(100*1024, 100*1024)
		c.Response().Header()["Content-Type"] = []string{"image/png"}
		_, err = io.Copy(c.Response(), ratelimit.Reader(f, b))
		return err
	})
...

If I changed url to the local test server:

url := "http://127.0.0.1:10233/test-http"

All 5 connections are used to download the file.
Output:

2020/08/21 10:44:49 4 started
2020/08/21 10:44:49 2 started
2020/08/21 10:44:49 0 started
2020/08/21 10:44:49 3 started
2020/08/21 10:44:49 1 started
2020/08/21 10:44:49 4 downloading map[Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:44:49 GMT] Vary:[Accept-Encoding]] -1 200 OK
2020/08/21 10:44:49 3 downloading map[Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:44:49 GMT] Vary:[Accept-Encoding]] -1 200 OK
2020/08/21 10:44:49 1 downloading map[Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:44:49 GMT] Vary:[Accept-Encoding]] -1 200 OK
2020/08/21 10:44:49 2 downloading map[Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:44:49 GMT] Vary:[Accept-Encoding]] -1 200 OK
2020/08/21 10:44:49 0 downloading map[Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:44:49 GMT] Vary:[Accept-Encoding]] -1 200 OK
2020/08/21 10:45:06 4 1894098 <nil>
2020/08/21 10:45:06 3 1894098 <nil>
2020/08/21 10:45:06 1 1894098 <nil>
2020/08/21 10:45:06 0 1894098 <nil>
2020/08/21 10:45:06 2 1894098 <nil>

image

URL https://fengshuai.oss-cn-shenzhen.aliyuncs.com/mainpage/background_home_main.png also seems fine:

[08.21 10:50:12] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:50:12] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:50:12] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:50:12] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:50:12] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:50:20] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 close, 679 bytes sent, 3394688 bytes (3.23 MB) received, lifetime 00:08
[08.21 10:50:21] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 close, 679 bytes sent, 3394688 bytes (3.23 MB) received, lifetime 00:09
[08.21 10:50:32] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 close, 679 bytes sent, 3394688 bytes (3.23 MB) received, lifetime 00:20
[08.21 10:50:32] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 close, 648 bytes sent, 3394688 bytes (3.23 MB) received, lifetime 00:20
[08.21 10:50:32] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 close, 648 bytes sent, 3394688 bytes (3.23 MB) received, lifetime 00:20
2020/08/21 10:50:12 4 started
2020/08/21 10:50:12 0 started
2020/08/21 10:50:12 1 started
2020/08/21 10:50:12 3 started
2020/08/21 10:50:12 2 started
2020/08/21 10:50:15 0 downloading map[Accept-Ranges:[bytes] Connection:[keep-alive] Content-Length:[3321320] Content-Md5:[LcF8uAHYES7A9R3ikeBiZQ==] Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:50:15 GMT] Etag:["2DC17CB801D8112EC0F51DE291E06265"] Last-Modified:[Tue, 02 Jun 2020 15:28:35 GMT] Server:[AliyunOSS] X-Oss-Hash-Crc64ecma:[627351119386910532] X-Oss-Object-Type:[Normal] X-Oss-Request-Id:[5F3F36673FD43B34366CD3EA] X-Oss-Server-Time:[9] X-Oss-Storage-Class:[Standard]] 3321320 200 OK
2020/08/21 10:50:15 3 downloading map[Accept-Ranges:[bytes] Connection:[keep-alive] Content-Length:[3321320] Content-Md5:[LcF8uAHYES7A9R3ikeBiZQ==] Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:50:15 GMT] Etag:["2DC17CB801D8112EC0F51DE291E06265"] Last-Modified:[Tue, 02 Jun 2020 15:28:35 GMT] Server:[AliyunOSS] X-Oss-Hash-Crc64ecma:[627351119386910532] X-Oss-Object-Type:[Normal] X-Oss-Request-Id:[5F3F366785D3EB3735B74BB5] X-Oss-Server-Time:[1] X-Oss-Storage-Class:[Standard]] 3321320 200 OK
2020/08/21 10:50:15 4 downloading map[Accept-Ranges:[bytes] Connection:[keep-alive] Content-Length:[3321320] Content-Md5:[LcF8uAHYES7A9R3ikeBiZQ==] Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:50:16 GMT] Etag:["2DC17CB801D8112EC0F51DE291E06265"] Last-Modified:[Tue, 02 Jun 2020 15:28:35 GMT] Server:[AliyunOSS] X-Oss-Hash-Crc64ecma:[627351119386910532] X-Oss-Object-Type:[Normal] X-Oss-Request-Id:[5F3F366888181632352091D8] X-Oss-Server-Time:[1] X-Oss-Storage-Class:[Standard]] 3321320 200 OK
2020/08/21 10:50:16 2 downloading map[Accept-Ranges:[bytes] Connection:[keep-alive] Content-Length:[3321320] Content-Md5:[LcF8uAHYES7A9R3ikeBiZQ==] Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:50:16 GMT] Etag:["2DC17CB801D8112EC0F51DE291E06265"] Last-Modified:[Tue, 02 Jun 2020 15:28:35 GMT] Server:[AliyunOSS] X-Oss-Hash-Crc64ecma:[627351119386910532] X-Oss-Object-Type:[Normal] X-Oss-Request-Id:[5F3F3668F0F9753636A214D4] X-Oss-Server-Time:[8] X-Oss-Storage-Class:[Standard]] 3321320 200 OK
2020/08/21 10:50:17 0 3321320 <nil>
2020/08/21 10:50:17 1 downloading map[Accept-Ranges:[bytes] Connection:[keep-alive] Content-Length:[3321320] Content-Md5:[LcF8uAHYES7A9R3ikeBiZQ==] Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:50:18 GMT] Etag:["2DC17CB801D8112EC0F51DE291E06265"] Last-Modified:[Tue, 02 Jun 2020 15:28:35 GMT] Server:[AliyunOSS] X-Oss-Hash-Crc64ecma:[627351119386910532] X-Oss-Object-Type:[Normal] X-Oss-Request-Id:[5F3F366A8338063437965BE9] X-Oss-Server-Time:[1] X-Oss-Storage-Class:[Standard]] 3321320 200 OK
2020/08/21 10:50:19 4 3321320 <nil>
2020/08/21 10:50:20 2 3321320 <nil>
2020/08/21 10:50:21 1 3321320 <nil>
2020/08/21 10:50:32 3 3321320 <nil>

I also tested i.imgur.com and pbs.twimg.com, all have same issue.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions