net/http: PUT: DefaultClient significantly slower than DefaultTransport when network is slow

[subbu05]

What version of Go are you using (go version)?
go version go1.10.3 darwin/amd64

Does this issue reproduce with the latest release?
yes

What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN="/Users/test/go/bin"
GOCACHE="/Users/test/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/test/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10.3/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10.3/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/76/trlnwqwj0bs5003x850kzpg00000gn/T/go-build781190647=/tmp/go-build -gno-record-gcc-switches -fno-common"

Scenario:
I have a network with Download speed :400kbit/s Upload:400kbit/s RTT:200ms
I have http client developed in Golang.
Wanted to transfer 1MB file to a web server.

What did you expect to see?
File upload must almost be same to curl or other web clients.

What did you see instead?
Curl took ~28s
Go client took ~42s

More Information on environment, tools:
How does Go react when there are issues with network.
To mimic network delay - https://github.com/sitespeedio/throttle

web server : https://github.com/sashgorokhov/docker-nginx-webdav

Source Code

package main

import (
  
   "io/ioutil"
   "time"
   "fmt"
   "net/http"
   "strings"
  
)

func main() {

   client := &http.Client{}
   url := "http://localhost:9080/file.txt"
   rb, _ := ioutil.ReadFile("file1mb.txt")
   request, err := http.NewRequest("PUT", url, strings.NewReader(string(rb)))
   //request.ContentLength = int64(len(string(rb)))
   sTime := time.Now()
   response, err := client.Do(request)
   eTime := time.Since(sTime)
   fmt.Println(eTime.String())
   if err != nil {
      fmt.Println(err)
   } else {
      fmt.Println(request.Header)
      defer response.Body.Close()
      fmt.Println(response.Status)
    }
}

Tried with and without Content-Length header(chunked encoding) in the http request.

My concern is for 1mb file I see a difference what if file size is 100MB+?

[subbu05]

For a 30MB file we saw difference of about 3minutes+ with Go net/http when compared to Curl.
Condition/Environment: When network is slow.

[agnivade]

Is this reproducible with client and server being on the same machine ? Would like to investigate this on my laptop.

[subbu05]

@agnivade Yes this is continuously reproducible, provided we delay the n/w.

[agnivade]

Oh, I see this is for http. Could you also kindly check for http2 client and see if you get similar results ?

This would help us pin-point the issue faster.

[subbu05]

I tested with http2 transport and there is a small difference between Go code and Curl.
Network Condition: Down:400kbit/s Up:400kbit/s RTT:200ms
Ex: 1MB file - Avg upload speed Curl ~25s Go ~29s
10MB - Avg upload speed Curl ~7m:30s Go ~7m:56s

[meirf]

@subbu05 have you had a chance to run the same code using http.DefaultTransport which is the core of the Client? That could help us narrow down where the bottleneck is.

[agnivade]

/cc @bradfitz

[subbu05]

@meirf I tested with http.DefaultTransport and I see curl and golang client file upload are all most ~same rate.

[meirf]

@subbu05 using your exact code as the client and the below Go code as the server, I am not seeing any difference in performance vs curl. Can you please show us the exact command you are using to run https://github.com/sashgorokhov/docker-nginx-webdav?

Go server:

package main

import (
	"fmt"
	"io"
	"net/http"
	"os"
	"log"
)

func main() {
	http.HandleFunc("/file.txt", uploadHandler)
	http.ListenAndServe(":9080", nil)
}

func uploadHandler(w http.ResponseWriter, r *http.Request) {
	log.Printf("uploadHandler start")
	file, err := os.Create("./result")
	if err != nil {
		panic(err)
	}
	n, err := io.Copy(file, r.Body)
	if err != nil {
		panic(err)
	}

	w.Write([]byte(fmt.Sprintf("%d bytes are recieved.\n", n)))
	log.Printf("uploadHandler end")
}

curl -X PUT -H "Content-Type: application/octet-stream" --data-binary '@file1mb.txt' http://127.0.0.1:9080/file.txt

[subbu05]

@meirf
Steps:
0. Install throttle - https://github.com/sitespeedio/throttle

1. Started the nginx container - docker run --name webdav -p 9080:80 -v /media:/media -d sashgorokhov/webdav
2. to mimic the latency in the network run the command throttle --profile 3gslow
3. run curl command - time curl -X PUT -T file.txt "http://localhost:9080/file1.txt"
4. run go client with same file.txt to the url http://localhost:9080/file2.txt (capture the time)

Observed:
30Mb file upload - I am seeing 3 minutes+ difference between curl and go.

[bradfitz]

@meirf, the title of this bug no longer makes sense, since DefaultClient uses DefaultTransport. I don't see how they could have different speeds.