url: Fix NO_PROXY env var to work properly with --proxy option. #1140

eramoto · 2016-11-25T05:40:14Z

The combination of --noproxy option and http_proxy env var works well
both for proxied hosts and non-proxied hosts.

However, when combining NO_PROXY env var with --proxy option,
non-proxied hosts are not reachable while proxied host is OK.

This patch allows us to access non-proxied hosts even if using NO_PROXY
env var with --proxy option.

mention-bot · 2016-11-25T05:40:20Z

@eramoto, thanks for your PR! By analyzing the history of the files in this pull request, we identified @bagder, @dfandrich and @rousskov to be potential reviewers.

bagder · 2016-11-27T10:49:58Z

allows us to access non-proxied hosts even if using NO_PROXY env var with --proxy option

It isn't very clear to me exactly how this works (after this patch). Are you saying that the proxy given with --proxy now is used even if the host in the URL is mentioned in the NO_PROXY environment variable?

http_proxy, no_proxy, --noproxy and --proxy all interact here. Do you think you can clarify how in the docs, like in the --proxy or --noproxy section?

notok · 2016-11-28T11:07:00Z

@bagder eramoto is saying that NO_PROXY environment variable is ignored when used with --proxy option.

So, for example, before this patch

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://somewhere.example.com/
# -> Accessed through proxy

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://192.168.1.1/
# -> Accessed through proxy ** NOT INTENDED **

after this patch

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://somewhere.example.com/
# -> Accessed through proxy

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://192.168.1.1/
# -> Accessed directly, as intended.

@eramoto please tell me if this is not what you meant.

bagder · 2016-11-29T14:03:41Z

That's a crystal clear explanation that I understand perfectly. @eramoto, do you think you could also add something to the docs in this vein that lets future users understand the priorities and what overrides what?

eramoto · 2016-12-02T08:31:28Z

@bagder

I add the following descriptions to the curl man-page:

--noproxy option does not override NO_PROXY env var.
If specify both of --noproxy option and NO_PROXY env var, both are effective.
Since 7.52.0, NO_PROXY env var is effective even if specify --proxy option.

Cloud you confirm this commit.

bagder · 2016-12-03T23:29:54Z

docs/curl.1

@@ -2312,6 +2317,16 @@ Sets the proxy server to use if no protocol-specific proxy is set.
 .IP "NO_PROXY <comma-separated list of hosts>"
 list of host names that shouldn't go through any proxy. If set to a asterisk
 \&'*' only, it matches all hosts.
+
+Since 7.52.0, this einvironment variable disable the proxy even if specify


Spelling: "environment"

bagder · 2016-12-03T23:30:08Z

docs/curl.1

@@ -2312,6 +2317,16 @@ Sets the proxy server to use if no protocol-specific proxy is set.
 .IP "NO_PROXY <comma-separated list of hosts>"
 list of host names that shouldn't go through any proxy. If set to a asterisk
 \&'*' only, it matches all hosts.
+
+Since 7.52.0, this einvironment variable disable the proxy even if specify
+\fI--porxy\fP option. That is


Spelling:" --proxy"

bagder

Just two spelling errors, apart from that I think it looks ready to merge!

eramoto · 2016-12-05T03:01:02Z

Thank you for your reviews. @bagder

I fixed these typos.

bagder · 2016-12-09T00:13:39Z

Do we really need the environment variable for NOPROXY to override the command line option --noproxy? It feels like against how several other options works. We generally let command line options override environment variables so that if you know what you want to can specify that exactly with command line options and if you rather want to let the environment control you don't use the command line option.

eramoto · 2016-12-14T08:35:22Z

Because it is better to change to the behavior that are similar to other options,
I now rewrite lib/url.c and docs/curl.1 to override the environment variable NO_PROXY
due to the command line option --noproxy.

Cloud you wait a few days. @bagder

eramoto · 2016-12-22T11:46:57Z

I changed as following:

The command line option --noproxy overrides the environment variable NO_PROXY.
Rewrite this behavior into the individual option documentation page noproxy.d.

Cloud you check this commits. @bagder

bagder · 2016-12-27T21:39:56Z

I thought we agreed that the command line options should override the environment variables? Yet your man page edit says:

this environment variable disable the proxy even if specify
--proxy option. That is
NO_PROXY=direct.example.com curl -x http://proxy.example.com http://direct.example.com
accesses the target URL directly.

notok · 2016-12-28T07:13:59Z

@bagder
I think an option should override the corresponding env var, like --noproxy overrides NO_PROXY,
but not for different env var, like --proxy(-x) option does not override NO_PROXY.

bagder · 2016-12-28T23:36:12Z

Hm, yes I think you're right. Thanks.

bagder · 2016-12-28T23:38:47Z

@eramoto , Can you please rebase your patch set (and squash what you think is reasonable), and update the release version number mentioned in the docs where this change is included (7.52.2 is now the next possible release number) and I promise to do my best to merge it when done?

bagder · 2016-12-28T23:39:46Z

(basically, you can skip changing curl.1 completely, we'll generate that based on the cmdline-opts/* docs anyway)

The combination of --noproxy option and http_proxy env var works well both for proxied hosts and non-proxied hosts. However, when combining NO_PROXY env var with --proxy option, non-proxied hosts are not reachable while proxied host is OK. This patch allows us to access non-proxied hosts even if using NO_PROXY env var with --proxy option.

If defined CURL_DISABLE_HTTP, detect_proxy() returned NULL. If not defined CURL_DISABLE_HTTP, detect_proxy() checked noproxy list. Thus refactor to set proxy to NULL instead of calling detect_proxy() if define CURL_DISABLE_HTTP, and refactor to call detect_proxy() if not define CURL_DISABLE_HTTP and the host is not in the noproxy list.

Under condition using http_proxy env var, noproxy list was the combination of --noproxy option and NO_PROXY env var previously. Since this commit, --noproxy option overrides NO_PROXY environment variable even if use http_proxy env var.

eramoto · 2017-01-10T04:41:26Z

I rebased and squashed my patch set (, and updated the release version number in the docs).

@bagder
Could you please merge this new patch set?

bagder · 2017-01-13T10:27:01Z

Thanks a lot for your work and patience on this!

This allows users to specify an override value for the noproxy list, in addition to overriding the proxy url, in proxy_info_from_url(). If no override is given, we should respect the values in the no_proxy/NO_PROXY environment variables, if present. This behavior can be seen in other tools like curl (see curl/curl#1140). I would also tweak this for the python3 module, but its ProxyInfo class doesn't even seem to have a bypass_hosts attribute.

bagder added the connecting & proxies label Nov 27, 2016

bagder reviewed Dec 3, 2016

View reviewed changes

bagder approved these changes Dec 3, 2016

View reviewed changes

eramoto added 3 commits January 6, 2017 21:48

eramoto force-pushed the fix-noproxy-env-with-proxy-option branch from c211af4 to 59fa2ed Compare January 6, 2017 13:01

bagder closed this in 2ac1942 Jan 13, 2017

bagder mentioned this pull request Jan 13, 2017

Please document syntax and semantics of NO_PROXY environment variable. #1208

Closed

houglum mentioned this pull request May 25, 2017

Respect NO_PROXY env var in proxy_info_from_url(). httplib2/httplib2#58

Merged

ehuss mentioned this pull request Oct 4, 2018

Explicitly configure noproxy support in Curl rust-lang/cargo#6115

Closed

lock bot locked as resolved and limited conversation to collaborators Jan 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

url: Fix NO_PROXY env var to work properly with --proxy option. #1140

url: Fix NO_PROXY env var to work properly with --proxy option. #1140

eramoto commented Nov 25, 2016

mention-bot commented Nov 25, 2016

bagder commented Nov 27, 2016

notok commented Nov 28, 2016

bagder commented Nov 29, 2016

eramoto commented Dec 2, 2016

bagder Dec 3, 2016

bagder Dec 3, 2016

bagder left a comment

eramoto commented Dec 5, 2016

bagder commented Dec 9, 2016 •

edited

Loading

eramoto commented Dec 14, 2016 •

edited

Loading

eramoto commented Dec 22, 2016

bagder commented Dec 27, 2016

notok commented Dec 28, 2016

bagder commented Dec 28, 2016

bagder commented Dec 28, 2016

bagder commented Dec 28, 2016

eramoto commented Jan 10, 2017

bagder commented Jan 13, 2017

url: Fix NO_PROXY env var to work properly with --proxy option. #1140

url: Fix NO_PROXY env var to work properly with --proxy option. #1140

Conversation

eramoto commented Nov 25, 2016

mention-bot commented Nov 25, 2016

bagder commented Nov 27, 2016

notok commented Nov 28, 2016

bagder commented Nov 29, 2016

eramoto commented Dec 2, 2016

bagder Dec 3, 2016

Choose a reason for hiding this comment

bagder Dec 3, 2016

Choose a reason for hiding this comment

bagder left a comment

Choose a reason for hiding this comment

eramoto commented Dec 5, 2016

bagder commented Dec 9, 2016 • edited Loading

eramoto commented Dec 14, 2016 • edited Loading

eramoto commented Dec 22, 2016

bagder commented Dec 27, 2016

notok commented Dec 28, 2016

bagder commented Dec 28, 2016

bagder commented Dec 28, 2016

bagder commented Dec 28, 2016

eramoto commented Jan 10, 2017

bagder commented Jan 13, 2017

bagder commented Dec 9, 2016 •

edited

Loading

eramoto commented Dec 14, 2016 •

edited

Loading