url: Fix NO_PROXY env var to work properly with --proxy option. #1140

Closed
wants to merge 3 commits into
from

Projects

None yet

4 participants

@eramoto
Contributor
eramoto commented Nov 25, 2016

The combination of --noproxy option and http_proxy env var works well
both for proxied hosts and non-proxied hosts.

However, when combining NO_PROXY env var with --proxy option,
non-proxied hosts are not reachable while proxied host is OK.

This patch allows us to access non-proxied hosts even if using NO_PROXY
env var with --proxy option.

@mention-bot

@eramoto, thanks for your PR! By analyzing the history of the files in this pull request, we identified @bagder, @dfandrich and @rousskov to be potential reviewers.

@bagder
Member
bagder commented Nov 27, 2016

allows us to access non-proxied hosts even if using NO_PROXY env var with --proxy option

It isn't very clear to me exactly how this works (after this patch). Are you saying that the proxy given with --proxy now is used even if the host in the URL is mentioned in the NO_PROXY environment variable?

http_proxy, no_proxy, --noproxy and --proxy all interact here. Do you think you can clarify how in the docs, like in the --proxy or --noproxy section?

@notok
notok commented Nov 28, 2016

@bagder eramoto is saying that NO_PROXY environment variable is ignored when used with --proxy option.

So, for example, before this patch

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://somewhere.example.com/
# -> Accessed through proxy

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://192.168.1.1/
# -> Accessed through proxy ** NOT INTENDED **

after this patch

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://somewhere.example.com/
# -> Accessed through proxy

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://192.168.1.1/
# -> Accessed directly, as intended.

@eramoto please tell me if this is not what you meant.

@bagder
Member
bagder commented Nov 29, 2016

That's a crystal clear explanation that I understand perfectly. @eramoto, do you think you could also add something to the docs in this vein that lets future users understand the priorities and what overrides what?

@eramoto
Contributor
eramoto commented Dec 2, 2016

@bagder

I add the following descriptions to the curl man-page:

  • --noproxy option does not override NO_PROXY env var.
  • If specify both of --noproxy option and NO_PROXY env var, both are effective.
  • Since 7.52.0, NO_PROXY env var is effective even if specify --proxy option.

Cloud you confirm this commit.

docs/curl.1
@@ -2312,6 +2317,16 @@ Sets the proxy server to use if no protocol-specific proxy is set.
.IP "NO_PROXY <comma-separated list of hosts>"
list of host names that shouldn't go through any proxy. If set to a asterisk
\&'*' only, it matches all hosts.
+
+Since 7.52.0, this einvironment variable disable the proxy even if specify
@bagder
bagder Dec 3, 2016 Member

Spelling: "environment"

docs/curl.1
@@ -2312,6 +2317,16 @@ Sets the proxy server to use if no protocol-specific proxy is set.
.IP "NO_PROXY <comma-separated list of hosts>"
list of host names that shouldn't go through any proxy. If set to a asterisk
\&'*' only, it matches all hosts.
+
+Since 7.52.0, this einvironment variable disable the proxy even if specify
+\fI--porxy\fP option. That is
@bagder
bagder Dec 3, 2016 Member

Spelling:" --proxy"

@bagder
bagder approved these changes Dec 3, 2016 View changes

Just two spelling errors, apart from that I think it looks ready to merge!

@eramoto
Contributor
eramoto commented Dec 5, 2016

Thank you for your reviews. @bagder

I fixed these typos.

@bagder
Member
bagder commented Dec 9, 2016 edited

Do we really need the environment variable for NOPROXY to override the command line option --noproxy? It feels like against how several other options works. We generally let command line options override environment variables so that if you know what you want to can specify that exactly with command line options and if you rather want to let the environment control you don't use the command line option.

@eramoto
Contributor
eramoto commented Dec 14, 2016 edited

Because it is better to change to the behavior that are similar to other options,
I now rewrite lib/url.c and docs/curl.1 to override the environment variable NO_PROXY
due to the command line option --noproxy.

Cloud you wait a few days. @bagder

@eramoto
Contributor
eramoto commented Dec 22, 2016

I changed as following:

  • The command line option --noproxy overrides the environment variable NO_PROXY.
  • Rewrite this behavior into the individual option documentation page noproxy.d.

Cloud you check this commits. @bagder

@bagder
Member
bagder commented Dec 27, 2016

I thought we agreed that the command line options should override the environment variables? Yet your man page edit says:

this environment variable disable the proxy even if specify
--proxy option. That is
NO_PROXY=direct.example.com curl -x http://proxy.example.com http://direct.example.com
accesses the target URL directly.

@notok
notok commented Dec 28, 2016

@bagder
I think an option should override the corresponding env var, like --noproxy overrides NO_PROXY,
but not for different env var, like --proxy(-x) option does not override NO_PROXY.

@bagder
Member
bagder commented Dec 28, 2016

Hm, yes I think you're right. Thanks.

@bagder
Member
bagder commented Dec 28, 2016

@eramoto , Can you please rebase your patch set (and squash what you think is reasonable), and update the release version number mentioned in the docs where this change is included (7.52.2 is now the next possible release number) and I promise to do my best to merge it when done?

@bagder
Member
bagder commented Dec 28, 2016

(basically, you can skip changing curl.1 completely, we'll generate that based on the cmdline-opts/* docs anyway)

eramoto added some commits Nov 25, 2016
@eramoto eramoto url: Fix NO_PROXY env var to work properly with --proxy option.
The combination of --noproxy option and http_proxy env var works well
both for proxied hosts and non-proxied hosts.

However, when combining NO_PROXY env var with --proxy option,
non-proxied hosts are not reachable while proxied host is OK.

This patch allows us to access non-proxied hosts even if using NO_PROXY
env var with --proxy option.
795306d
@eramoto eramoto url: Refactor detect_proxy()
If defined CURL_DISABLE_HTTP, detect_proxy() returned NULL. If not
defined CURL_DISABLE_HTTP, detect_proxy() checked noproxy list.

Thus refactor to set proxy to NULL instead of calling detect_proxy() if
define CURL_DISABLE_HTTP, and refactor to call detect_proxy() if not
define CURL_DISABLE_HTTP and the host is not in the noproxy list.
b6722df
@eramoto eramoto url: --noproxy option overrides NO_PROXY environment variable
Under condition using http_proxy env var, noproxy list was the combination
of --noproxy option and NO_PROXY env var previously. Since this commit,
--noproxy option overrides NO_PROXY environment variable even if use
http_proxy env var.
59fa2ed
@eramoto
Contributor
eramoto commented Jan 10, 2017

I rebased and squashed my patch set (, and updated the release version number in the docs).

@bagder
Could you please merge this new patch set?

@bagder bagder added a commit that closed this pull request Jan 13, 2017
@eramoto @bagder eramoto + bagder url: --noproxy option overrides NO_PROXY environment variable
Under condition using http_proxy env var, noproxy list was the
combination of --noproxy option and NO_PROXY env var previously. Since
this commit, --noproxy option overrides NO_PROXY environment variable
even if use http_proxy env var.

Closes #1140
2ac1942
@bagder bagder closed this in 2ac1942 Jan 13, 2017
@bagder
Member
bagder commented Jan 13, 2017

Thanks a lot for your work and patience on this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment