New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
net/http: considering supporting CIDR notation in no_proxy env variable #16704
Comments
I'd rather not extend the functionality of You can implement |
@bradfitz afaics the entire use of NO_PROXY and proxy env vars is effectively convention, and afaics.. the issue with transport construction, is that as go based software becomes more popular its not even an option for the users to set that up themselves, their dependent on std lib behavior. ie. here's a recent debug session i had where curl and python respect it but a go based binary (etcdctl) does not. this makes quite a bit painful for users of go software, and i'm seeing it filed multiple times on software written in go (particularly docker/moby).
curl works with it
as does python
as does ruby
but go based binaries don't
|
@kapilt, okay, I'll reopen, but I don't plan to work on this myself. |
For this to move forward, sometime should try to first write a "spec" for it, documenting which other languages/libraries already implement said spec. (But if languages Foo and Bar both just use libcurl or whatever, say that.) |
Looking at implementations across other languages and browsers, a full implementation of a no_proxy spec can get complicated quite quickly, and there are arguments about what is right and what is wrong. I vote to keep it simple with domain, ip and CIDR exclusions. This is what python requests library does. My need for no_proxy in corporate and government environments is just excluding domains and private subnets ( Ports, URL schemes, IP wildcards Following simple rules:
References python requests library (very simple - domain, ip, CIDR check) Chromium (more complex - support for different schemes, ports and parts of domains) Some more reading |
CL https://golang.org/cl/47853 mentions this issue. |
I'd like to help out on this so I created the above proposal and hopefully, it get's approved and then I can get started on the coding side with a little nudge in the right direction |
@bradfitz Is there anything else that needs to be done to consider this for inclusion in a future release? I like many others are hitting issues with Go programs not recognizing CIDR ranges in the NO_PROXY while other applications/languages accept them. One has to become creative when specifying a NO_PROXY to work around the limitations of any Go program. |
I'm still on leave. /cc @tombergan |
For golang/go#16704. Change-Id: Id718d290628d9a1e723f7df0434ded30c3f08e02 Reviewed-on: https://go-review.googlesource.com/47853 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
I'm back from leave. I've submitted your proposal doc CL. Visible at https://github.com/golang/proposal/blob/master/design/16704-cidr-notation-no-proxy.md What's not entirely clear from your doc is who all supports CIDRs in $no_proxy. Just Python? Or others? This seems fine, though. Want to prepare a CL? It won't happen for Go 1.10, though, which entered feature freeze today. |
@forrejam's comment has a nice algorithm summary and answers the question of who supports CIDRs in no_proxy. @kapilt's comment has another summary. Looks like chrome, firefox, libproxy, python, ruby, and curl support CIDRs in $no_proxy. |
@tombergan, ah, right. I had expected that to be summarized in the doc, though. In any case, I'm fine with this. Somebody can send a CL. |
Change https://golang.org/cl/75730 mentions this issue: |
@bradfitz Given your last comment in my change, should I just abandon this change and create a new one for x/net/http/proxy? |
Change https://golang.org/cl/68091 mentions this issue: |
From x/net git rev c7086645de2. Updates #16704 Change-Id: I4d642478fc69a52c973964845fca2fd402716e57 Reviewed-on: https://go-review.googlesource.com/68091 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
Change https://golang.org/cl/115255 mentions this issue: |
I've hit this problem too - another +1 for an upstream fix. (We might work around it ourselves for now.) |
Found this thread while investigating Packer issues (compiled in GoLang) in which it is implied a fix may go out in 1.11 due this month? Hope so. At any rate let me +1 this as well. Proxy use in certain verticals is common, and in the age of automation we often find ourselves needing to hit IPs as DNS simply isn't available yet. |
From what I can see in https://go-review.googlesource.com/c/net/+/115255 it looks like it is progressing! :) |
Someone please fix this. I wasted a lifetime figuring this out. Thank you :-) |
NO_PROXY includes support for CIDR, and notations can also match exactly on port information if provided. When specifying a port with IPv6, the address must be enclosed with square brackets, [IPv6 address]:port. Updates golang/go#16704 (fixes after vendor into std) Change-Id: Ideb61a9ec60a6b1908f5a2c885cd6d9dd10c37cf Reviewed-on: https://go-review.googlesource.com/115255 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Change https://golang.org/cl/122655 mentions this issue: |
Change https://golang.org/cl/122619 mentions this issue: |
Feature request - It would be helpful if the net/http client supported CIDR notation in the no_proxy environment variable, or allowed another way to configure the net/http client to ignore the http_proxy for large networks (ie 10.0.0.0/8 etc).
I had no luck finding an official "spec" for the no_proxy environment variable, but I believe supporting this notation would cause no issues and would allow users of net/http to add proxy exclusions for large networks.
Python requests library implements this feature with the no_proxy env variable - https://github.com/kennethreitz/requests/blob/master/requests/utils.py#L569
and most browsers allow some sort of CIDR notation / wildcards for proxy exclusions.
During my searching, I came across an old issue #2158 where the functionality was discussed but not really addressed (as far as the CIDR notation goes)
go version
)?go version go1.6.3 linux/amd64
go env
)?GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
Tried to use CIDR notation in the no_proxy env variable to exclude net/http client from using web proxy specified in http_proxy env variable for private network 10.0.0.0/8
Expected http client library to ignore http_proxy for addresses in the 10.0.0.0/8 network.
net/http client tried to use the http_proxy for get/post requests to addresses in 10.0.0.0/8 network.
Cheers, and thanks for golang!
The text was updated successfully, but these errors were encountered: