Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: improve proxy handling #37623

Open
vlourme opened this issue Mar 3, 2020 · 3 comments
Open

net/http: improve proxy handling #37623

vlourme opened this issue Mar 3, 2020 · 3 comments
Milestone

Comments

@vlourme
Copy link

@vlourme vlourme commented Mar 3, 2020

What version of Go are you using (go version)?

$ go version
go version go1.14 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/vico/Library/Caches/go-build"
GOENV="/Users/vico/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/vico/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/32/7qmymd5950b3nhv4dd7qbqkw0000gn/T/go-build036564932=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I'm running requests (using go-resty library, based on default HTTP client). This problem is I need to use proxies, some of proxies are dead (public proxies) and so, they don't answer the request.
I started learning Go not a long time ago, but when I was programming in C#, the proxy handling was good, I mean :

  • Most of proxies respond well. Here in Go, the proxy definition is sketchy (in my opinion). There is also no official way to put proxy with authentication (not sure?).
  • When a proxy is dead, it throws an error fast. Here in Go, it waits for the default page timeout, not a connect timeout or something.

What did you expect to see?

Proxy integration that respond better or that skips wrong/dead proxies faster.

What did you see instead?

Mainly performance losses, and when I use public proxies, Go seems not connecting to all proxies while some are really working.

(Original issue from go-resty: go-resty/resty#320)

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Mar 3, 2020

Is this issue possible to reproduce? It would be helpful to provide a snippet that can reproduce it, then it's easier to investigate this issue.

@dmitshur dmitshur changed the title Improve proxy handling net/http: improve proxy handling Mar 3, 2020
@vlourme
Copy link
Author

@vlourme vlourme commented Mar 3, 2020

Here is a sample of how I use proxies:

client := http.Client{
	Transport: &http.Transport{
		Proxy: func(request *http.Request) (url *url.URL, err error) {
			// Wrong proxy
			return url.Parse("http://103.69.216.125:8080")
		},
	},
	Timeout: 10 * time.Second,
}

resp, err := client.Get("https://httpbin.org/get")

if err != nil {
	fmt.Println(err)
	return
}

fmt.Printf("Status code: %d", resp.StatusCode)

I put voluntarily a free proxy (from spys.ru) which had a bad uptime (maybe the proxy is dead).
So here the response is : Get "https://httpbin.org/get": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The thing is, it waits 10 seconds to return that the proxy is bad.
Why not improving the proxy testing, instead of waiting the client timeout to throw the error?

The exemple here may seem a little silly. But when I use a lot of proxies and I run a good amount of requests in multiples workers, when a proxy is down, it slow down everything by waiting the page timeout.

Maybe should we implement a timeout to test proxy before (I don't know how we could do that).
And also, I propose to create a better proxy implementation (if possible), which could looks like:

// Create a proxy
myProxy := proxy.New(protocol, host, port)
	
// Optionally, putting a authentication
myProxy.setAuthentication(username, password)
	 
// Then linking our proxy to our transport
transport := http.Transport{
	Proxy: myProxy,
}
	
// Or to the client directly, which could result in gain of time 
// if we don't have other settings to edit in the transport
client.setProxy(myProxy)

I'd like to get a thing like this, it looks more structured than just putting our proxy as parsed URL to the transport.

@dmitshur dmitshur added this to the Backlog milestone Mar 3, 2020
@dmitshur
Copy link
Member

@dmitshur dmitshur commented Mar 3, 2020

Thanks for providing more information.

/cc @bradfitz @rsc per owners.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.