Closed
Description
What version of Go are you using (go version
)?
$ go version go version go1.15 windows/amd64
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env
)?
go env
Output
$ go env set GO111MODULE= set GOARCH=amd64 set GOBIN= set GOCACHE=C:\Users\user\AppData\Local\go-build set GOENV=C:\Users\user\AppData\Roaming\go\env set GOEXE=.exe set GOFLAGS= set GOHOSTARCH=amd64 set GOHOSTOS=windows set GOINSECURE= set GOMODCACHE=C:\Users\user\go\pkg\mod set GONOPROXY= set GONOSUMDB= set GOOS=windows set GOPATH=C:\Users\user\go set GOPRIVATE= set GOPROXY=https://proxy.golang.org,direct set GOROOT=c:\go set GOSUMDB=sum.golang.org set GOTMPDIR= set GOTOOLDIR=c:\go\pkg\tool\windows_amd64 set GCCGO=gccgo set AR=ar set CC=gcc set CXX=g++ set CGO_ENABLED=1 set GOMOD= set CGO_CFLAGS=-g -O2 set CGO_CPPFLAGS= set CGO_CXXFLAGS=-g -O2 set CGO_FFLAGS=-g -O2 set CGO_LDFLAGS=-g -O2 set PKG_CONFIG=pkg-config set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\user\AppData\Local\Temp\go-build545182214=/tmp/go-build -gno-record-gcc-switches
What did you do?
I am downloading images from i.pximg.net
with multiple goroutines.
My code:
package main
import (
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"sync"
)
func main() {
wg := sync.WaitGroup{}
client := &http.Client{Transport: &http.Transport{}}
for i := 0; i < 5; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
log.Println(i, "started")
url := fmt.Sprintf("https://i.pximg.net/img-original/img/2019/09/13/00/14/39/76752268_p%d.jpg", i)
// url := "http://127.0.0.1/test-http"
req, err := http.NewRequest("GET", url, nil)
// Set the Referer header so we will not get 403 error
req.Header["Referer"] = []string{"https://www.pixiv.net/"}
res, err := client.Do(req)
if err != nil {
log.Println(err)
return
}
defer res.Body.Close()
log.Println(i, "downloading", res.Header, res.ContentLength, res.Status)
wr, err := io.Copy(ioutil.Discard, res.Body)
log.Println(i, wr, err)
}(i)
}
wg.Wait()
}
What did you expect to see?
The client will create 5 connections to the server and download the content.
What did you see instead?
The client created 5 connections to the server at first, but 4 connections closed immediately. Only one connection was used to transfer the data.
Output:
2020/08/21 10:46:25 4 started
2020/08/21 10:46:25 3 started
2020/08/21 10:46:25 1 started
2020/08/21 10:46:25 2 started
2020/08/21 10:46:25 0 started
2020/08/21 10:46:27 1 downloading map[Accept-Ranges:[bytes] Age:[136306] Cache-Control:[max-age=31536000] Content-Length:[618736] Content-Type:[image/jpeg] Date:[Fri, 21 Aug 2020 02:46:28 GMT] Expires:[Thu, 19 Aug 2021 12:54:42 GMT] Last-Modified:[Thu, 12 Sep 2019 15:14:39 GMT] Server:[nginx] Via:[http/1.1 f002 (second)] X-Content-Type-Options:[nosniff]] 618736 200 OK
2020/08/21 10:46:28 4 downloading map[Accept-Ranges:[bytes] Age:[136248] Cache-Control:[max-age=31536000] Content-Length:[651153] Content-Type:[image/jpeg] Date:[Fri, 21 Aug 2020 02:46:28 GMT] Expires:[Thu, 19 Aug 2021 12:55:40 GMT] Last-Modified:[Thu, 12 Sep 2019 15:14:39 GMT] Server:[nginx] Via:[http/1.1 f001 (second)] X-Content-Type-Options:[nosniff]] 651153 200 OK
2020/08/21 10:46:28 2 downloading map[Accept-Ranges:[bytes] Age:[163397] Cache-Control:[max-age=31536000] Content-Length:[683337] Content-Type:[image/jpeg] Date:[Fri, 21 Aug 2020 02:46:28 GMT] Expires:[Thu, 19 Aug 2021 05:23:11 GMT] Last-Modified:[Thu, 12 Sep 2019 15:14:39 GMT] Server:[nginx] Via:[http/1.1 f001 (second)] X-Content-Type-Options:[nosniff]] 683337 200 OK
2020/08/21 10:46:28 3 downloading map[Accept-Ranges:[bytes] Age:[1255] Cache-Control:[max-age=31536000] Content-Length:[594356] Content-Type:[image/jpeg] Date:[Fri, 21 Aug 2020 02:46:28 GMT] Expires:[Sat, 21 Aug 2021 02:25:33 GMT] Last-Modified:[Thu, 12 Sep 2019 15:14:39 GMT] Server:[nginx]
Via:[http/1.1 f005 (second)] X-Content-Type-Options:[nosniff]] 594356 200 OK
2020/08/21 10:46:28 0 downloading map[Accept-Ranges:[bytes] Age:[175766] Cache-Control:[max-age=31536000] Content-Length:[5681832] Content-Type:[image/jpeg] Date:[Fri, 21 Aug 2020 02:46:28 GMT] Expires:[Thu, 19 Aug 2021 01:57:02 GMT] Last-Modified:[Thu, 12 Sep 2019 15:14:39 GMT] Server:[nginx] Via:[http/1.1 f003 (second)] X-Content-Type-Options:[nosniff]] 5681832 200 OK
2020/08/21 10:46:29 1 618736 <nil>
2020/08/21 10:46:29 4 651153 <nil>
2020/08/21 10:46:29 2 683337 <nil>
2020/08/21 10:46:29 3 594356 <nil>
2020/08/21 10:46:31 0 5681832 <nil>
Proxifier log (An app which can force other apps to use a proxy)
[08.21 10:25:31] main.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:25:31] main.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:25:31] main.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:25:31] main.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:25:31] main.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:25:32] main.exe *64 - i.pximg.net:443 close, 438 bytes sent, 4860 bytes (4.74 KB) received, lifetime 00:01
[08.21 10:25:32] main.exe *64 - i.pximg.net:443 close, 438 bytes sent, 4860 bytes (4.74 KB) received, lifetime 00:01
[08.21 10:25:32] main.exe *64 - i.pximg.net:443 close, 438 bytes sent, 4860 bytes (4.74 KB) received, lifetime 00:01
[08.21 10:25:32] main.exe *64 - i.pximg.net:443 close, 438 bytes sent, 4860 bytes (4.74 KB) received, lifetime 00:01
[08.21 10:25:43] main.exe *64 - i.pximg.net:443 close, 42961 bytes (41.9 KB) sent, 8263702 bytes (7.88 MB) received, lifetime 00:12
I tried to build a local server with echo
to test:
...
e.GET("/test-http", func(c echo.Context) error {
f, err := os.Open(`test.png`)
if err != nil {
return err
}
defer f.Close()
b := ratelimit.NewBucketWithRate(100*1024, 100*1024)
c.Response().Header()["Content-Type"] = []string{"image/png"}
_, err = io.Copy(c.Response(), ratelimit.Reader(f, b))
return err
})
...
If I changed url
to the local test server:
url := "http://127.0.0.1:10233/test-http"
All 5 connections are used to download the file.
Output:
2020/08/21 10:44:49 4 started
2020/08/21 10:44:49 2 started
2020/08/21 10:44:49 0 started
2020/08/21 10:44:49 3 started
2020/08/21 10:44:49 1 started
2020/08/21 10:44:49 4 downloading map[Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:44:49 GMT] Vary:[Accept-Encoding]] -1 200 OK
2020/08/21 10:44:49 3 downloading map[Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:44:49 GMT] Vary:[Accept-Encoding]] -1 200 OK
2020/08/21 10:44:49 1 downloading map[Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:44:49 GMT] Vary:[Accept-Encoding]] -1 200 OK
2020/08/21 10:44:49 2 downloading map[Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:44:49 GMT] Vary:[Accept-Encoding]] -1 200 OK
2020/08/21 10:44:49 0 downloading map[Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:44:49 GMT] Vary:[Accept-Encoding]] -1 200 OK
2020/08/21 10:45:06 4 1894098 <nil>
2020/08/21 10:45:06 3 1894098 <nil>
2020/08/21 10:45:06 1 1894098 <nil>
2020/08/21 10:45:06 0 1894098 <nil>
2020/08/21 10:45:06 2 1894098 <nil>
URL https://fengshuai.oss-cn-shenzhen.aliyuncs.com/mainpage/background_home_main.png
also seems fine:
[08.21 10:50:12] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:50:12] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:50:12] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:50:12] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:50:12] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 open through proxy 127.0.0.1:1081 SOCKS5
[08.21 10:50:20] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 close, 679 bytes sent, 3394688 bytes (3.23 MB) received, lifetime 00:08
[08.21 10:50:21] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 close, 679 bytes sent, 3394688 bytes (3.23 MB) received, lifetime 00:09
[08.21 10:50:32] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 close, 679 bytes sent, 3394688 bytes (3.23 MB) received, lifetime 00:20
[08.21 10:50:32] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 close, 648 bytes sent, 3394688 bytes (3.23 MB) received, lifetime 00:20
[08.21 10:50:32] main.exe *64 - fengshuai.oss-cn-shenzhen.aliyuncs.com:443 close, 648 bytes sent, 3394688 bytes (3.23 MB) received, lifetime 00:20
2020/08/21 10:50:12 4 started
2020/08/21 10:50:12 0 started
2020/08/21 10:50:12 1 started
2020/08/21 10:50:12 3 started
2020/08/21 10:50:12 2 started
2020/08/21 10:50:15 0 downloading map[Accept-Ranges:[bytes] Connection:[keep-alive] Content-Length:[3321320] Content-Md5:[LcF8uAHYES7A9R3ikeBiZQ==] Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:50:15 GMT] Etag:["2DC17CB801D8112EC0F51DE291E06265"] Last-Modified:[Tue, 02 Jun 2020 15:28:35 GMT] Server:[AliyunOSS] X-Oss-Hash-Crc64ecma:[627351119386910532] X-Oss-Object-Type:[Normal] X-Oss-Request-Id:[5F3F36673FD43B34366CD3EA] X-Oss-Server-Time:[9] X-Oss-Storage-Class:[Standard]] 3321320 200 OK
2020/08/21 10:50:15 3 downloading map[Accept-Ranges:[bytes] Connection:[keep-alive] Content-Length:[3321320] Content-Md5:[LcF8uAHYES7A9R3ikeBiZQ==] Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:50:15 GMT] Etag:["2DC17CB801D8112EC0F51DE291E06265"] Last-Modified:[Tue, 02 Jun 2020 15:28:35 GMT] Server:[AliyunOSS] X-Oss-Hash-Crc64ecma:[627351119386910532] X-Oss-Object-Type:[Normal] X-Oss-Request-Id:[5F3F366785D3EB3735B74BB5] X-Oss-Server-Time:[1] X-Oss-Storage-Class:[Standard]] 3321320 200 OK
2020/08/21 10:50:15 4 downloading map[Accept-Ranges:[bytes] Connection:[keep-alive] Content-Length:[3321320] Content-Md5:[LcF8uAHYES7A9R3ikeBiZQ==] Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:50:16 GMT] Etag:["2DC17CB801D8112EC0F51DE291E06265"] Last-Modified:[Tue, 02 Jun 2020 15:28:35 GMT] Server:[AliyunOSS] X-Oss-Hash-Crc64ecma:[627351119386910532] X-Oss-Object-Type:[Normal] X-Oss-Request-Id:[5F3F366888181632352091D8] X-Oss-Server-Time:[1] X-Oss-Storage-Class:[Standard]] 3321320 200 OK
2020/08/21 10:50:16 2 downloading map[Accept-Ranges:[bytes] Connection:[keep-alive] Content-Length:[3321320] Content-Md5:[LcF8uAHYES7A9R3ikeBiZQ==] Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:50:16 GMT] Etag:["2DC17CB801D8112EC0F51DE291E06265"] Last-Modified:[Tue, 02 Jun 2020 15:28:35 GMT] Server:[AliyunOSS] X-Oss-Hash-Crc64ecma:[627351119386910532] X-Oss-Object-Type:[Normal] X-Oss-Request-Id:[5F3F3668F0F9753636A214D4] X-Oss-Server-Time:[8] X-Oss-Storage-Class:[Standard]] 3321320 200 OK
2020/08/21 10:50:17 0 3321320 <nil>
2020/08/21 10:50:17 1 downloading map[Accept-Ranges:[bytes] Connection:[keep-alive] Content-Length:[3321320] Content-Md5:[LcF8uAHYES7A9R3ikeBiZQ==] Content-Type:[image/png] Date:[Fri, 21 Aug 2020 02:50:18 GMT] Etag:["2DC17CB801D8112EC0F51DE291E06265"] Last-Modified:[Tue, 02 Jun 2020 15:28:35 GMT] Server:[AliyunOSS] X-Oss-Hash-Crc64ecma:[627351119386910532] X-Oss-Object-Type:[Normal] X-Oss-Request-Id:[5F3F366A8338063437965BE9] X-Oss-Server-Time:[1] X-Oss-Storage-Class:[Standard]] 3321320 200 OK
2020/08/21 10:50:19 4 3321320 <nil>
2020/08/21 10:50:20 2 3321320 <nil>
2020/08/21 10:50:21 1 3321320 <nil>
2020/08/21 10:50:32 3 3321320 <nil>
I also tested i.imgur.com
and pbs.twimg.com
, all have same issue.