net/http: no_proxy does not handle numeric IP addr/range

[gopherbot]

by Paul.A.Lalonde:

I live behind a gloriously huge firewall and proxy server.  All internal IP addresses,
on 10.0.0.0/8 should avoid the proxy server, as they don't get redirected inwards.

Normally, adding '10.0.0.0/8' to my no_proxy environment variable handles this.

Reading "http/transport.go:/^func useProxy" it's evident that the code only
handles names or literal numeric matches.  Instead, the code should resolve the input
address and match against any numeric proxies (including subnet masking).  It should
probably also do a name lookup on any numeric input addresses and match them against
string names in no_proxy.

What steps will reproduce the problem?
1. Proxy all your traffic  (set HTTP_PROXY to some proxy server)
    - Try to connect to any local go web server using your *external* facing IP address (not localhost).  This should timeout/503.

2. Make an exception for your local domain (set NO_PROXY to include 192.168.0.0/16, for
example, if you're behind a local router)
    - Try to connect again.  This will still fail, instead of succeeding.

Which compiler are you using (5g, 6g, 8g, gccgo)?
6g


Which operating system are you using?
Linux Ubuntu 10.10


Which revision are you using?  (hg identify)
d2b506b47343+ tip

[rsc]

Comment 1:

Thanks.  This will be easy to fix once we have the new Dial in place.

Owner changed to @bradfitz.

Status changed to Accepted.

[gopherbot]

Comment 2 by Paul.A.Lalonde:

I totally botched the repro conditions - I should have let it sit over lunch.
It is, of course, only when you open the connection from Go, using Get or equivalent,
that the proxies get invoked, which is where my app is having grief.  I'll add a
reproducer shortly.

[gopherbot]

Comment 3 by Paul.A.Lalonde:

Here's a quick modification of pkg/http/proxy_test.go that at least includes some cases
I care about.  I chose "stable" external addresses to lookup.
A question: is the new Dial a 1 week horizon, or a multi-month horizon?  For the later
I'll throw some extra logic into my user code; for the former I'll grin and bear it :-)
Thanks,
    Paul

Attachments:

1. proxy_test.go (1720 bytes)

[rsc]

Comment 4:

probably closer to the latter.

[bradfitz]

Comment 5:

Now that we have ParseCIDR, this is less work than it was before:
http://tip.goneat.org/pkg/net/#IP.ParseCIDR
But what are the rules for the "HTTP_PROXY" environment variable?  Is there a spec of
any sort?
Do we really have to do the hostname DNS lookup first before deciding whether or not to
use a proxy?  That is, if "foo.intranet" resolves to 10.2.3.4, it should trigger for
10.0.0.0/8?
I just want to do whatever other browsers do.

Status changed to WaitingForReply.

[gopherbot]

Comment 6 by Paul.A.Lalonde:

http://www.w3.org/Daemon/User/Proxies/ProxyClients.html looks like the most
authoritative resource.
Paul

[bradfitz]

Comment 7:

Well, I think we comply with that spec already:
"Some clients support the no_proxy environment variable that specifies a set of domains
for which the proxy should not be consulted; the contents is a comma-separated list of
domain names, with an optional :port part:
        no_proxy="cern.ch,ncsa.uiuc.edu,some.host:8080"
    export no_proxy
"
Let's try to find something written past July 1995.  :)

[gopherbot]

Comment 8 by Paul.A.Lalonde:

I should have added that "authoritative" doesn't include "complete".  
no_proxy including subnets/CIDR notation is something I got from Chrome, I think.
Paul

[bradfitz]

Comment 9:

[+evan@chromium.org, Chrome guy who writes Go and might have answers]

[evmar]

Comment 10:

I went to check Chrome's code to see what no_proxy meant, but my code search turned up
Python and Perl's code first.  Maybe you'll find all of them helpful:
http://codesearch.google.com/codesearch#search/&exact_package=chromium&q=no_proxy&type=cs
Here's the relevant Chrome code:
http://codesearch.google.com/codesearch#OAMlx_jo-ck/src/net/proxy/proxy_config_service_linux.cc&exact_package=chromium&ct=rc&cd=13&q=no_proxy&sq=&l=179
It appears there is code for special semantics of no_proxy.
I don't have enough of a checkout at the moment to look at the blame.
PS. Proxy settings are a huge complex mess.  People always yell at us for not using
libproxy which knows how to e.g. obey gconf and evaluate JavaScript for PAC files.

[rsc]

Comment 11:

Hey, you know what would make any HTTP client better?
Needing to evaluate arbitrary JavaScript code to decide
how to connect to a host.  :-)
Russ

[bradfitz]

Comment 12:

I knew there was a reason for the Go V8 bindings.  But I think we need to put Nigel onto
a pure Go port for the standard library.  :)

[rsc]

Comment 13:

Labels changed: added priority-later.

[rsc]

Comment 14:

Status changed to Accepted.

[rsc]

Comment 15:

Labels changed: added go1.1.