NO_PROXY CIDRs not handled properly

[Gmerold]

Hi,
I've stumbled upon an interesting issue with httpcore, when running behind the proxy. It turns out that httpcore doesn't handle CIDRs from NO_PROXY correctly.
Here's my NO_PROXY list:

NO_PROXY: 127.0.0.1,localhost,.local,.svc,10.134.0.0/16,10.134.248.191/24,10.152.183.0/24

Here's how httpcore sees it:

{'http://': Proxy('http:/MY_PROXY_IP:PORT'), 'https://': Proxy('MY_PROXY_IP:PORT'), 'all://127.0.0.1': None, 'all://localhost': None, 'all://*.local': None, 'all://*.svc': None, 'all://10.134.0.0/16': None, 'all://10.134.248.191/24': None, 'all://10.152.183.0/24': None}

And with that config, here's the URL object built for my request:

URL(scheme=b'http', host=b'MY_PROXY_IP', port=PORT, target=b'10.152.183.1:443')

The target IP falls into one of the CIDRs listed in NO_PROXY, and yet, httpcore uses proxy for it.
As a result I'm getting below traceback:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-test-0/charm/venv/httpx/_transports/default.py", line 63, in map_httpcore_exceptions
    yield
  File "/var/lib/juju/agents/unit-test-0/charm/venv/httpx/_transports/default.py", line 226, in handle_request
    resp = self._pool.handle_request(req)
  File "/var/lib/juju/agents/unit-test-0/charm/venv/httpcore/_sync/connection_pool.py", line 262, in handle_request
    raise exc
  File "/var/lib/juju/agents/unit-test-0/charm/venv/httpcore/_sync/connection_pool.py", line 245, in handle_request
    response = connection.handle_request(request)
  File "/var/lib/juju/agents/unit-test-0/charm/venv/httpcore/_sync/http_proxy.py", line 285, in handle_request
    raise ProxyError(msg)
httpcore.ProxyError: 403 Forbidden

If I put 10.152.183.1 explicitly in the NO_PROXY list, everything works fine.

I suppose it is a case for an issue. Please let me know if I should go ahead and create one.

Cheers!

[karpetrosyan]

Hello, the proxy routing is handled by httpx, so we should first move this to the correct location.

[Gmerold]

Good point. Thanks for moving it.

[phvalguima]

Hi, indeed this is an issue and it is caused by httpx, thanks @karosis88 @Gmerold. Here is a quick reproducer:

If I have the following configuration, where my NO_PROXY has value: 10.165.186.0/24, as follows:

# NO_PROXY="10.165.186.0/24" HTTPS_PROXY="http://10.165.186.1:8080" HTTP_PROXY="http://10.165.186.1:8080" python3
on 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
 "help", "copyright", "credits" or "license" for more information.
from httpx._client import Client
from httpx._models import Request
test=Client()
test.send(Request("GET", "http://10.165.186.46:8080/"))

I'd expect that IP 10.165.186.46 to be reached directly, as it is part of NO_PROXY's CIDR. However, I can see it hits my proxy instead.

I've written a quick change in URLPattern that seems to fix this problem: phvalguima@a90801b

[karpetrosyan]

HTTPX simply does not support the ip ranges. Instead of 10.165.186.0/24 in your case simply use 10.165.186.0.
See also: #2737

Example

Shell

export NO_PROXY="10.165.186.0/24"

Python

from httpx._client import Client

test = Client()

for pattern, value in test._mounts.items():
    print(pattern.pattern, value)

OUTPUT

all://10.165.186.0/24 None

It just thinks that 24 is a url path.

[Gmerold]

Hi,
Thanks for the clarification.
What about the fix proposed by @phvalguima? Seems reasonable to me. IP ranges are broadly used in such cases (in my case it's a Kubernetes subnet) and it would be great to support those. Listing all IPs of a /16 subnet seems a little cumbersome ;)

[karpetrosyan]

Answered in #2737.