net/http: improve proxy handling

[vlourme]

What version of Go are you using (go version)?

$ go version
go version go1.14 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/vico/Library/Caches/go-build"
GOENV="/Users/vico/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/vico/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/32/7qmymd5950b3nhv4dd7qbqkw0000gn/T/go-build036564932=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I'm running requests (using go-resty library, based on default HTTP client). This problem is I need to use proxies, some of proxies are dead (public proxies) and so, they don't answer the request.
I started learning Go not a long time ago, but when I was programming in C#, the proxy handling was good, I mean :

• Most of proxies respond well. Here in Go, the proxy definition is sketchy (in my opinion). There is also no official way to put proxy with authentication (not sure?).
• When a proxy is dead, it throws an error fast. Here in Go, it waits for the default page timeout, not a connect timeout or something.

What did you expect to see?

Proxy integration that respond better or that skips wrong/dead proxies faster.

What did you see instead?

Mainly performance losses, and when I use public proxies, Go seems not connecting to all proxies while some are really working.

(Original issue from go-resty: go-resty/resty#320)

[dmitshur]

Is this issue possible to reproduce? It would be helpful to provide a snippet that can reproduce it, then it's easier to investigate this issue.

[vlourme]

Here is a sample of how I use proxies:

client := http.Client{
	Transport: &http.Transport{
		Proxy: func(request *http.Request) (url *url.URL, err error) {
			// Wrong proxy
			return url.Parse("http://103.69.216.125:8080")
		},
	},
	Timeout: 10 * time.Second,
}

resp, err := client.Get("https://httpbin.org/get")

if err != nil {
	fmt.Println(err)
	return
}

fmt.Printf("Status code: %d", resp.StatusCode)

I put voluntarily a free proxy (from spys.ru) which had a bad uptime (maybe the proxy is dead).
So here the response is : Get "https://httpbin.org/get": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The thing is, it waits 10 seconds to return that the proxy is bad.
Why not improving the proxy testing, instead of waiting the client timeout to throw the error?

The exemple here may seem a little silly. But when I use a lot of proxies and I run a good amount of requests in multiples workers, when a proxy is down, it slow down everything by waiting the page timeout.

Maybe should we implement a timeout to test proxy before (I don't know how we could do that).
And also, I propose to create a better proxy implementation (if possible), which could looks like:

// Create a proxy
myProxy := proxy.New(protocol, host, port)
	
// Optionally, putting a authentication
myProxy.setAuthentication(username, password)
	 
// Then linking our proxy to our transport
transport := http.Transport{
	Proxy: myProxy,
}
	
// Or to the client directly, which could result in gain of time 
// if we don't have other settings to edit in the transport
client.setProxy(myProxy)

I'd like to get a thing like this, it looks more structured than just putting our proxy as parsed URL to the transport.

[dmitshur]

Thanks for providing more information.

/cc @bradfitz @rsc per owners.