aiohttp not honoring `no_proxy`

[scirelli]

Long story short

aiohttp not honoring no_proxy environment variable when trust_env is set.

Expected behaviour

When there is a no_proxy NO_PROXY environment variable set, URLs listed in that variable should not use http{s}_proxy

Actual behaviour

When trust_env is set and there are http_proxy and no_proxy set, the proxy is always used.

Steps to reproduce

Set http{s}_proxy and no_proxy, make a request to a domain listed in no_proxy. The request will still go through the proxy.

Your environment

Linux
aiohttp 3.6.2
Company proxy server

[asvetlov]

The pull request is welcome!

[scirelli]

Could you point me in the right direction?

I'm guessing this is where I'll need to check the no proxy list https://github.com/aio-libs/aiohttp/blob/master/aiohttp/client.py#L424 and blank the proxy if found?

[asvetlov]

Hmm, interesting.
The real code for extracting the proxy list from env vars is proxies_from_env() function.

It doesn't parse itself but reuses getproxies() from urllib.request.
I did not investigate more.
Does getproxies() ignore NO_PROXY?

[scirelli]

No. That is why I suggesting doing it manually. I could only find a reference to no_proxy when urllib.request.ProxyHandler is used. I'm searching for something to compare the hostnames to the no_proxy list.

I could add a is_in_no_proxy_list(url) to the helpers.py module.

[scirelli]

Maybe a solution like 5d92cb5 might work. Would still have to write the tests. I should probably read up on the no_proxy list standards to make sure hostnames put in the no_proxy list are matching correctly.

[scirelli]

Actually I looks like urllib has code to handle no_proxy. I'll use the appropriate function from there.

Any ETA for this? We're suffering from the same problem with aiohttp not honoring no_proxy vars.

[lbg-raghu-vennam]

Even I am interested in this fix as a third party application we are using is dependent on this library.

[paulbastian]

We tested these changes and they worked for us!

[scattym]

Glad I found this bug report! Just wondering if there are any workarounds until this patch gets accepted?

I need to use a proxy to get to external services but that proxy has no access to other internal services, hence my need for no_proxy.

At the moment the only option I seem to have is to create separate sessions with and without proxies set, but that means my app would need to be aware of the network setup of where it gets installed, which is certainly not ideal. Unless there are other options I am missing?