New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

url: Fix NO_PROXY env var to work properly with --proxy option. #1140

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
4 participants
@eramoto
Contributor

eramoto commented Nov 25, 2016

The combination of --noproxy option and http_proxy env var works well
both for proxied hosts and non-proxied hosts.

However, when combining NO_PROXY env var with --proxy option,
non-proxied hosts are not reachable while proxied host is OK.

This patch allows us to access non-proxied hosts even if using NO_PROXY
env var with --proxy option.

@mention-bot

This comment has been minimized.

mention-bot commented Nov 25, 2016

@eramoto, thanks for your PR! By analyzing the history of the files in this pull request, we identified @bagder, @dfandrich and @rousskov to be potential reviewers.

@bagder

This comment has been minimized.

Member

bagder commented Nov 27, 2016

allows us to access non-proxied hosts even if using NO_PROXY env var with --proxy option

It isn't very clear to me exactly how this works (after this patch). Are you saying that the proxy given with --proxy now is used even if the host in the URL is mentioned in the NO_PROXY environment variable?

http_proxy, no_proxy, --noproxy and --proxy all interact here. Do you think you can clarify how in the docs, like in the --proxy or --noproxy section?

@notok

This comment has been minimized.

notok commented Nov 28, 2016

@bagder eramoto is saying that NO_PROXY environment variable is ignored when used with --proxy option.

So, for example, before this patch

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://somewhere.example.com/
# -> Accessed through proxy

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://192.168.1.1/
# -> Accessed through proxy ** NOT INTENDED **

after this patch

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://somewhere.example.com/
# -> Accessed through proxy

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://192.168.1.1/
# -> Accessed directly, as intended.

@eramoto please tell me if this is not what you meant.

@bagder

This comment has been minimized.

Member

bagder commented Nov 29, 2016

That's a crystal clear explanation that I understand perfectly. @eramoto, do you think you could also add something to the docs in this vein that lets future users understand the priorities and what overrides what?

@eramoto

This comment has been minimized.

Contributor

eramoto commented Dec 2, 2016

@bagder

I add the following descriptions to the curl man-page:

  • --noproxy option does not override NO_PROXY env var.
  • If specify both of --noproxy option and NO_PROXY env var, both are effective.
  • Since 7.52.0, NO_PROXY env var is effective even if specify --proxy option.

Cloud you confirm this commit.

docs/curl.1 Outdated
@@ -2312,6 +2317,16 @@ Sets the proxy server to use if no protocol-specific proxy is set.
.IP "NO_PROXY <comma-separated list of hosts>"
list of host names that shouldn't go through any proxy. If set to a asterisk
\&'*' only, it matches all hosts.
Since 7.52.0, this einvironment variable disable the proxy even if specify

This comment has been minimized.

@bagder

bagder Dec 3, 2016

Member

Spelling: "environment"

docs/curl.1 Outdated
@@ -2312,6 +2317,16 @@ Sets the proxy server to use if no protocol-specific proxy is set.
.IP "NO_PROXY <comma-separated list of hosts>"
list of host names that shouldn't go through any proxy. If set to a asterisk
\&'*' only, it matches all hosts.
Since 7.52.0, this einvironment variable disable the proxy even if specify
\fI--porxy\fP option. That is

This comment has been minimized.

@bagder

bagder Dec 3, 2016

Member

Spelling:" --proxy"

@bagder

bagder approved these changes Dec 3, 2016

Just two spelling errors, apart from that I think it looks ready to merge!

@eramoto

This comment has been minimized.

Contributor

eramoto commented Dec 5, 2016

Thank you for your reviews. @bagder

I fixed these typos.

@bagder

This comment has been minimized.

Member

bagder commented Dec 9, 2016

Do we really need the environment variable for NOPROXY to override the command line option --noproxy? It feels like against how several other options works. We generally let command line options override environment variables so that if you know what you want to can specify that exactly with command line options and if you rather want to let the environment control you don't use the command line option.

@eramoto

This comment has been minimized.

Contributor

eramoto commented Dec 14, 2016

Because it is better to change to the behavior that are similar to other options,
I now rewrite lib/url.c and docs/curl.1 to override the environment variable NO_PROXY
due to the command line option --noproxy.

Cloud you wait a few days. @bagder

@eramoto

This comment has been minimized.

Contributor

eramoto commented Dec 22, 2016

I changed as following:

  • The command line option --noproxy overrides the environment variable NO_PROXY.
  • Rewrite this behavior into the individual option documentation page noproxy.d.

Cloud you check this commits. @bagder

@bagder

This comment has been minimized.

Member

bagder commented Dec 27, 2016

I thought we agreed that the command line options should override the environment variables? Yet your man page edit says:

this environment variable disable the proxy even if specify
--proxy option. That is
NO_PROXY=direct.example.com curl -x http://proxy.example.com http://direct.example.com
accesses the target URL directly.

@notok

This comment has been minimized.

notok commented Dec 28, 2016

@bagder
I think an option should override the corresponding env var, like --noproxy overrides NO_PROXY,
but not for different env var, like --proxy(-x) option does not override NO_PROXY.

@bagder

This comment has been minimized.

Member

bagder commented Dec 28, 2016

Hm, yes I think you're right. Thanks.

@bagder

This comment has been minimized.

Member

bagder commented Dec 28, 2016

@eramoto , Can you please rebase your patch set (and squash what you think is reasonable), and update the release version number mentioned in the docs where this change is included (7.52.2 is now the next possible release number) and I promise to do my best to merge it when done?

@bagder

This comment has been minimized.

Member

bagder commented Dec 28, 2016

(basically, you can skip changing curl.1 completely, we'll generate that based on the cmdline-opts/* docs anyway)

eramoto added some commits Nov 25, 2016

url: Fix NO_PROXY env var to work properly with --proxy option.
The combination of --noproxy option and http_proxy env var works well
both for proxied hosts and non-proxied hosts.

However, when combining NO_PROXY env var with --proxy option,
non-proxied hosts are not reachable while proxied host is OK.

This patch allows us to access non-proxied hosts even if using NO_PROXY
env var with --proxy option.
url: Refactor detect_proxy()
If defined CURL_DISABLE_HTTP, detect_proxy() returned NULL. If not
defined CURL_DISABLE_HTTP, detect_proxy() checked noproxy list.

Thus refactor to set proxy to NULL instead of calling detect_proxy() if
define CURL_DISABLE_HTTP, and refactor to call detect_proxy() if not
define CURL_DISABLE_HTTP and the host is not in the noproxy list.
url: --noproxy option overrides NO_PROXY environment variable
Under condition using http_proxy env var, noproxy list was the combination
of --noproxy option and NO_PROXY env var previously. Since this commit,
--noproxy option overrides NO_PROXY environment variable even if use
http_proxy env var.

@eramoto eramoto force-pushed the eramoto:fix-noproxy-env-with-proxy-option branch from c211af4 to 59fa2ed Jan 6, 2017

@eramoto

This comment has been minimized.

Contributor

eramoto commented Jan 10, 2017

I rebased and squashed my patch set (, and updated the release version number in the docs).

@bagder
Could you please merge this new patch set?

@bagder bagder closed this in 2ac1942 Jan 13, 2017

@bagder

This comment has been minimized.

Member

bagder commented Jan 13, 2017

Thanks a lot for your work and patience on this!

houglum added a commit to houglum/httplib2 that referenced this pull request May 25, 2017

Respect NO_PROXY env var in proxy_info_from_url().
This allows users to specify an override value for the noproxy list, in
addition to overriding the proxy url, in proxy_info_from_url(). If no
override is given, we should respect the values in the no_proxy/NO_PROXY
environment variables, if present. This behavior can be seen in other
tools like curl (see curl/curl#1140).

I would also tweak this for the python3 module, but its ProxyInfo class
doesn't even seem to have a bypass_hosts attribute.

galvez added a commit to httplib2/httplib2 that referenced this pull request Jun 21, 2017

Respect NO_PROXY env var in proxy_info_from_url(). (#58)
This allows users to specify an override value for the noproxy list, in
addition to overriding the proxy url, in proxy_info_from_url(). If no
override is given, we should respect the values in the no_proxy/NO_PROXY
environment variables, if present. This behavior can be seen in other
tools like curl (see curl/curl#1140).

I would also tweak this for the python3 module, but its ProxyInfo class
doesn't even seem to have a bypass_hosts attribute.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment