no/no_proxy is not honoured

[Suika]

I guess PRs are overlooked without an Issues. It's about #5596 and the way handle no_proxy. Since urllib handles no_proxy properly, it's the logic in requests that messes with the env in a seemingly twisted way.

Expected Result

The ability to use no_proxy vairable via OS and function arguments.

Actual Result

Only OS no_proxy ENV is being processed. Uncer certain conditions that were described multiple times in the no_proxy issues.

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.9.2"
  },
  "idna": {
    "version": "2.9"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.5"
  },
  "platform": {
    "release": "4.15.18-10-pve",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "1010107f",
    "version": "19.1.0"
  },
  "requests": {
    "version": "2.23.0"
  },
  "system_ssl": {
    "version": "1010107f"
  },
  "urllib3": {
    "version": "1.25.9"
  },
  "using_pyopenssl": true
}

[RichieB2B]

I have the same issue: no_proxy is ignored in a simple requests.get() call:

import requests

proxies = {
  'http': 'proxy.example.com',
  'no_proxy': 'google.com'
}

requests.get('http://google.com/', proxies=proxies)

With 2.28.0 this yields:

requests.exceptions.ProxyError: HTTPConnectionPool(host='proxy.example.com', port=80): Max retries exceeded with url: http://google.com/ (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10418f5b0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')))

Set the following in bash:

export http_proxy="proxy.example.com"
export no_proxy="google.com"

and the requests.get('http://google.com/') works just fine.

[sigmavirus24]

Where is it documented that what you believe is a big should work? At no point has this library supported what you've made up

[RichieB2B]

From the 2.14.0 release history "Improvements" section:

- It is now possible to pass ``no_proxy`` as a key to the ``proxies`` dictionary to provide handling similar to the ``NO_PROXY`` environment variable.

[sigmavirus24]

So something from ages ago that is likely not documented elsewhere?

[RichieB2B]

It all comes down to feature request #2817 and the implementation 85400d8 not actually working. If you think this is not a bug I'd happily create a documentation PR instead.

[RichieB2B]

I did some more testing and was able to make an exception using the per-host proxy settings:

proxies = {
  'http': 'http://proxy.example.com',
  'http://google.com': '',
}

This is more flexible than the no_proxy mechanism so I can live with it not working as described.

I've made a PR to clarify this in the documentation at #6172

[vimagick]

no_proxy doesn't work

python3 -c 'import requests; print(requests.get("http://ipinfo.io/ip", proxies={"no_proxy":"ipinfo.io","http":"http://myproxy:3128"}).text)'

no_proxy works

export no_proxy=ipinfo.io
export http_proxy=http://myproxy:3128

python3 -c 'import requests; print(requests.get("http://ipinfo.io/ip").text)'

---

The difference is caused by this if statement:

        if "proxies" not in kwargs:
            kwargs["proxies"] = resolve_proxies(request, self.proxies, self.trust_env)

no_proxy is handled by resolve_proxies()

https://github.com/psf/requests/blob/main/src/requests/sessions.py#L683-L684

[smarie]

EDIT: I mistakenly thought that this was working in 2.25.0, but this was because in 2.25.0 the send operation was not even using the OS env settings at all (as opposed to the post function). I therefore wonder if the following analysis is even relevant ?

It still seems from the code that the no_proxy variable is not read when a send (and not a post) is performed. This is because resolve_proxies (where the issue is) is only used in send and not in post. post relies on request, which as opposed to send, uses merge_environment_settings for proxy resolution and not resolve_proxies. This is not very consistent, maybe something to improve ?

---

I ran into this today

Only OS no_proxy ENV is being processed.

Actually, in requests==2.31.0 the OS env variable no_proxy is not processed correctly anymore. It was working correctly in 2.25.0 though, I just checked. So this is a regression. It seems to be related to the fact that once get_environ_proxies() is called, the no_proxy part of its contents is not returned.

requests/src/requests/utils.py

Lines 876 to 882 in 4f3f189

	if trust_env and not should_bypass_proxies(url, no_proxy=no_proxy):
	environ_proxies = get_environ_proxies(url, no_proxy=no_proxy)
	proxy = environ_proxies.get(scheme, environ_proxies.get("all"))
	if proxy:
	new_proxies.setdefault(scheme, proxy)

You can see that new_proxies

• is updated with the entry in environ_proxies corresponding to the scheme,
• but is not updated with the no_proxy contents if any in environ_proxies

Should I open a separate issue ?