New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: PUT: DefaultClient significantly slower than DefaultTransport when network is slow #26702

Open
subbu05 opened this Issue Jul 30, 2018 · 11 comments

Comments

Projects
None yet
4 participants
@subbu05

subbu05 commented Jul 30, 2018

What version of Go are you using (go version)?
go version go1.10.3 darwin/amd64

Does this issue reproduce with the latest release?
yes

What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN="/Users/test/go/bin"
GOCACHE="/Users/test/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/test/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10.3/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10.3/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/76/trlnwqwj0bs5003x850kzpg00000gn/T/go-build781190647=/tmp/go-build -gno-record-gcc-switches -fno-common"

Scenario:
I have a network with Download speed :400kbit/s Upload:400kbit/s RTT:200ms
I have http client developed in Golang.
Wanted to transfer 1MB file to a web server.

What did you expect to see?
File upload must almost be same to curl or other web clients.

What did you see instead?
Curl took ~28s
Go client took ~42s

More Information on environment, tools:
How does Go react when there are issues with network.
To mimic network delay - https://github.com/sitespeedio/throttle

web server : https://github.com/sashgorokhov/docker-nginx-webdav

Source Code

package main

import (
  
   "io/ioutil"
   "time"
   "fmt"
   "net/http"
   "strings"
  
)

func main() {

   client := &http.Client{}
   url := "http://localhost:9080/file.txt"
   rb, _ := ioutil.ReadFile("file1mb.txt")
   request, err := http.NewRequest("PUT", url, strings.NewReader(string(rb)))
   //request.ContentLength = int64(len(string(rb)))
   sTime := time.Now()
   response, err := client.Do(request)
   eTime := time.Since(sTime)
   fmt.Println(eTime.String())
   if err != nil {
      fmt.Println(err)
   } else {
      fmt.Println(request.Header)
      defer response.Body.Close()
      fmt.Println(response.Status)
    }
}

Tried with and without Content-Length header(chunked encoding) in the http request.

My concern is for 1mb file I see a difference what if file size is 100MB+?

@subbu05

This comment has been minimized.

subbu05 commented Jul 30, 2018

For a 30MB file we saw difference of about 3minutes+ with Go net/http when compared to Curl.
Condition/Environment: When network is slow.

@meirf meirf changed the title from File upload using http PUT is more slower than to CURL when network is slow to net/http: File PUT is slower than CURL when network is slow Jul 31, 2018

@meirf meirf added the Performance label Jul 31, 2018

@agnivade

This comment has been minimized.

Member

agnivade commented Jul 31, 2018

Is this reproducible with client and server being on the same machine ? Would like to investigate this on my laptop.

@agnivade agnivade added this to the Unplanned milestone Jul 31, 2018

@subbu05

This comment has been minimized.

subbu05 commented Jul 31, 2018

@agnivade Yes this is continuously reproducible, provided we delay the n/w.

@agnivade

This comment has been minimized.

Member

agnivade commented Jul 31, 2018

Oh, I see this is for http. Could you also kindly check for http2 client and see if you get similar results ?

This would help us pin-point the issue faster.

@subbu05

This comment has been minimized.

subbu05 commented Jul 31, 2018

I tested with http2 transport and there is a small difference between Go code and Curl.
Network Condition: Down:400kbit/s Up:400kbit/s RTT:200ms
Ex: 1MB file - Avg upload speed Curl ~25s Go ~29s
10MB - Avg upload speed Curl ~7m:30s Go ~7m:56s

@meirf

This comment has been minimized.

Member

meirf commented Aug 1, 2018

@subbu05 have you had a chance to run the same code using http.DefaultTransport which is the core of the Client? That could help us narrow down where the bottleneck is.

@agnivade

This comment has been minimized.

Member

agnivade commented Aug 1, 2018

/cc @bradfitz

@subbu05

This comment has been minimized.

subbu05 commented Aug 1, 2018

@meirf I tested with http.DefaultTransport and I see curl and golang client file upload are all most ~same rate.

@meirf meirf changed the title from net/http: File PUT is slower than CURL when network is slow to net/http: PUT: DefaultClient significantly slower than DefaultTransport when network is slow Aug 2, 2018

@meirf

This comment has been minimized.

Member

meirf commented Aug 4, 2018

@subbu05 using your exact code as the client and the below Go code as the server, I am not seeing any difference in performance vs curl. Can you please show us the exact command you are using to run https://github.com/sashgorokhov/docker-nginx-webdav?

Go server:

package main

import (
	"fmt"
	"io"
	"net/http"
	"os"
	"log"
)

func main() {
	http.HandleFunc("/file.txt", uploadHandler)
	http.ListenAndServe(":9080", nil)
}

func uploadHandler(w http.ResponseWriter, r *http.Request) {
	log.Printf("uploadHandler start")
	file, err := os.Create("./result")
	if err != nil {
		panic(err)
	}
	n, err := io.Copy(file, r.Body)
	if err != nil {
		panic(err)
	}

	w.Write([]byte(fmt.Sprintf("%d bytes are recieved.\n", n)))
	log.Printf("uploadHandler end")
}

curl -X PUT -H "Content-Type: application/octet-stream" --data-binary '@file1mb.txt' http://127.0.0.1:9080/file.txt

@subbu05

This comment has been minimized.

subbu05 commented Aug 6, 2018

@meirf
Steps:
0. Install throttle - https://github.com/sitespeedio/throttle

  1. Started the nginx container - docker run --name webdav -p 9080:80 -v /media:/media -d sashgorokhov/webdav
  2. to mimic the latency in the network run the command throttle --profile 3gslow
  3. run curl command - time curl -X PUT -T file.txt "http://localhost:9080/file1.txt"
  4. run go client with same file.txt to the url http://localhost:9080/file2.txt (capture the time)

Observed:
30Mb file upload - I am seeing 3 minutes+ difference between curl and go.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Nov 21, 2018

@meirf, the title of this bug no longer makes sense, since DefaultClient uses DefaultTransport. I don't see how they could have different speeds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment