Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: ProxyFromEnvironment does not support socks5 #18508

Closed
bruceauyeung opened this issue Jan 4, 2017 · 24 comments

Comments

Projects
None yet
10 participants
@bruceauyeung
Copy link

commented Jan 4, 2017

What version of Go are you using (go version)?

go version go1.7.4 linux/amd64

What operating system and processor architecture are you using (go env)?

actually it's Manjaro

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/bruce/go_ext_path"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build134643851=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

git config --global http.proxy 'socks5://127.0.0.1:1080'
git config --global https.proxy 'socks5://127.0.0.1:1080'
export http_proxy=socks5://127.0.0.1:1080
export https_proxy=socks5://127.0.0.1:1080
go get -u -v github.com/zmb3/gogetdoc

What did you expect to see?

Git can use socks5 proxy to clone a repo, so i expect go get can use socks5 proxy to download/update package github.com/zmb3/gogetdoc

What did you see instead?

github.com/zmb3/gogetdoc (download)
Fetching https://golang.org/x/tools/go/buildutil?go-get=1
https fetch failed: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout
package golang.org/x/tools/go/buildutil: unrecognized import path "golang.org/x/tools/go/buildutil" (https fetch: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout)
Fetching https://golang.org/x/tools/go/loader?go-get=1
https fetch failed: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout
package golang.org/x/tools/go/loader: unrecognized import path "golang.org/x/tools/go/loader" (https fetch: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout)
[bruce@bruce-manjaro ~]$ set https_proxy=socks5://127.0.0.1:1080
[bruce@bruce-manjaro ~]$ go get -u -v github.com/zmb3/gogetdoc
github.com/zmb3/gogetdoc (download)
Fetching https://golang.org/x/tools/go/buildutil?go-get=1
https fetch failed: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout
package golang.org/x/tools/go/buildutil: unrecognized import path "golang.org/x/tools/go/buildutil" (https fetch: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout)
Fetching https://golang.org/x/tools/go/loader?go-get=1
https fetch failed: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout
package golang.org/x/tools/go/loader: unrecognized import path "golang.org/x/tools/go/loader" (https fetch: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout)

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Jan 4, 2017

@josharian

This comment has been minimized.

Copy link
Contributor

commented Jan 4, 2017

From a quick glance, looks like maybe cmd/go needs to use http.ProxyFromEnvironment.

@josharian josharian changed the title go does not recognize socks5 proxy cmd/go: get does not use socks5 proxy Jan 4, 2017

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Jan 4, 2017

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Jan 4, 2017

@bruceauyeung

This comment has been minimized.

Copy link
Author

commented Jan 4, 2017

i tried to find if there is a spec for http.proxy, but found nothing.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jan 4, 2017

Yes, we don't support SOCK5 proxies in the standard library and thus we don't support it in the cmd/go tool.

We have SOCK5 support elsewhere, so we could vendor it into std at least for use by cmd/go, but it's not clear how much it matters. Is SOCK5 still widely used?

@bruceauyeung

This comment has been minimized.

Copy link
Author

commented Jan 4, 2017

@davecheney useful informations that i can found about http.proxy is http://unix.stackexchange.com/a/212972

@bradfitz In China, shadowsocks is used widely

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Jan 4, 2017

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jan 4, 2017

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jan 4, 2017

cmd/go does use http.ProxyFromEnvironment, because http.DefaultTransport does.

http.ProxyFromEnvironment does not support socks5. @bradfitz, where is the socks5 code we have elsewhere? It seems like it should be a tiny amount of code (under 100 lines plus tests) to proxy TCP connections into socks5. Assuming it really is small, I would not object to putting that into the standard library, on by default. It's very useful combined with ssh -D. I used to use that and tsocks, but Go doesn't play well with tsocks if I remember correctly, and tsocks is a bit of a hack.

As @davecheney says, at the least we should give a better error. But it would be nice to add support.

@rsc rsc added this to the Go1.9Early milestone Jan 4, 2017

@rsc rsc changed the title cmd/go: get does not use socks5 proxy net/http: ProxyFromEnvironment does not support socks5 Jan 4, 2017

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jan 4, 2017

@bradfitz bradfitz added the NeedsFix label Jan 4, 2017

@iamzhout

This comment has been minimized.

Copy link

commented Jan 5, 2017

@bruceauyeung Just suggest to use privoxy / squid alike tool to setup a local http proxy server over socks5 one to get workaround of this issue.

@bradfitz it would be useful if socks proxy could be supported. As to me it would also be ok if not as for the workaround I mentioned above.

@bruceauyeung

This comment has been minimized.

Copy link
Author

commented Jan 5, 2017

@bruceauyeung Just suggest to use privoxy / squid alike tool to setup a local http proxy server over socks5 one to get workaround of this issue.

@iamzhout thanks for your tip. now i go get packages through a http proxy (XX-Net), although it's slow.

@iamzhout

This comment has been minimized.

Copy link

commented Jan 5, 2017

@bradfitz actually there is another issue #18519 which really bothers me a lot. Hope there would be a fix or workaround also.

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Jan 5, 2017

@iamzhout it looks like your employer is MITM'ing your SSL communications.

Go get is just a wrapper over git, I recommend git cloning that repository directly.

@iamzhout

This comment has been minimized.

Copy link

commented Jan 5, 2017

@bruceauyeung, I do suggest you to give privoxy/squid a try, as the speed should be the same as to your socks5 one. and the configuration is really simple, take privoxy as an example, you just add below line to your /etc/privoxy/config file, and then do service privoxy restart, all done.

listen-address  0.0.0.0:12345
forward-socks5  /  your-socks5-ip:port  .
@icexin

This comment has been minimized.

Copy link

commented Jan 12, 2017

@bruceauyeung You may want this github.com/icexin/sockhttp. It's a small tool written in go that makes a HTTP proxy over SOCK5.

@bruceauyeung

This comment has been minimized.

Copy link
Author

commented Jan 12, 2017

@icexin i'll give it a shot, thanks for your work!

@mattn

This comment has been minimized.

Copy link
Member

commented Jan 13, 2017

Maybe, implementaion will be like below.

package main

import (
	"context"
	"io"
	"log"
	"net"
	"net/http"
	"os"

	"golang.org/x/net/proxy"
)

type dialer struct {
	addr   string
	socks5 proxy.Dialer
}

func (d *dialer) DialContext(ctx context.Context, network, addr string) (net.Conn, error) {
	// TODO: golang.org/x/net/proxy need to add DialContext
	return d.Dial(network, addr)
}

func (d *dialer) Dial(network, addr string) (net.Conn, error) {
	var err error
	if d.socks5 == nil {
		d.socks5, err = proxy.SOCKS5("tcp", d.addr, nil, proxy.Direct)
		if err != nil {
			return nil, err
		}
	}
	return d.socks5.Dial(network, addr)
}

func Socks5Proxy(addr string) *http.Transport {
	d := &dialer{addr: addr}
	return &http.Transport{
		DialContext: d.DialContext,
		Dial:        d.Dial,
	}
}

func main() {
	http.DefaultTransport = Socks5Proxy("127.0.0.1:1080")

	resp, err := http.Get("http://www.google.com/")
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()
	io.Copy(os.Stdout, resp.Body)
}

tested with server written in go https://github.com/armon/go-socks5

@mattn

This comment has been minimized.

Copy link
Member

commented Jan 13, 2017

So, to add common interfaces between http_proxy and socks5, I guess we should not modify ProxyFromEnvironment. it's better to add new API for set-up proxy.

@walken-google

This comment has been minimized.

Copy link
Contributor

commented Jan 18, 2017

I could use this feature too, and I've been looking into implementing it.

This has probably been answered before, but how do I go about importing a golang.org package (such as golang_org/x/net/proxy) so that the net/http package can use it ? I know this would require creating a src/vendor/golang_org/x/net/proxy directory but I'm not sure if it needs any tracking metadata, if the import needs to be a separate commit, etc...

@walken-google walken-google self-assigned this Jan 21, 2017

@walken-google

This comment has been minimized.

Copy link
Contributor

commented Jan 21, 2017

@gopherbot

This comment has been minimized.

Copy link

commented Jan 23, 2017

CL https://golang.org/cl/35488 mentions this issue.

gopherbot pushed a commit that referenced this issue Mar 2, 2017

net/http: add support for socks5 proxy
See #18508

This commit adds http Client support for socks5 proxies.

Change-Id: Ib015f3819801da13781d5acdd780149ae1f5857b
Reviewed-on: https://go-review.googlesource.com/35488
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Aug 9, 2017

Change https://golang.org/cl/41031 mentions this issue: net/http: replace SOCKS client implementation

gopherbot pushed a commit that referenced this issue Apr 6, 2018

net/http: replace SOCKS client implementation
This change replaces the vendored socks client implementation with the
bundle of golang.org/x/net/internal/socks package which contains fixes
for 19354 and 11682.

golang.org/x/net/internal/socks becomes socks_bundle.go.

At git rev 61147c4. (golang.org/cl/38278)

Updates #11682.
Updates #18508.
Updates #19354.
Fixes #19688.
Updates #21333.

Change-Id: I8cf6c3f5eb87c24685a7592be015729f84fbed77
Reviewed-on: https://go-review.googlesource.com/41031
Run-TryBot: Mikio Hara <mikioh.mikioh@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

@golang golang locked and limited conversation to collaborators Aug 9, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.