HTTPTransport doesn't detect HTTPS_PROXY

[wesbarnett]

I'm using another Python library that utilizes httpx to fetch a remote resource. The code they use is of the form (as a minimal example):

import httpx

url = "http://example.com"
with httpx.Client(transport=httpx.HTTPTransport()) as client:
    print(client.get(url))

This fails when utilizing a proxy, even though I have HTTPS_PROXY set, with the error:

httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known

The following variations do work (HTTPTransport isn't used or, if it is used, the proxy is set manually):

import httpx
import os

url = "http://example.com"

# variation 1
print(httpx.get(url))

# variation 2
with httpx.Client() as client:
    print(client.get(url))

# variation 3
with httpx.Client(transport=httpx.HTTPTransport(proxy=httpx.Proxy(os.environ["HTTPS_PROXY"]))) as client:
    print(client.get(url))

[tomchristie]

Heya Wes.

That's right, yup. An HTTPTransport instance won't use a proxy unless it's explicitly configured to use one.

When you use httpx.Client() you'll end up with one or more HTTPTransport instances mounted, depending on the HTTP_PROXY and HTTPS_PROXY environment variables. (Which might each configure zero, one, or several different proxies.)

So, suppose you had:

HTTPS_PROXY="http://127.0.0.1:1234"

Then you'd end up with the equivalent of...

client = httpx.Client(
    mounts={
        "https://": httpx.HTTPTransport(proxy=httpx.Proxy("http://127.0.0.1:1234")),
        "http://": httpx.HTTPTransport(),
    }
)

There's more complex cases if NO_PROXY is used too.

I'm not quite sure that I know how best to document/explain this, but transport instances either use a proxy or don't, so we don't apply the environment settings at that level because they're more complexly expressive that just ON/OFF. Instead the settings apply at the level of the client, which can then mount one or more correctly configured transports.

I'd be curious to understand more about why they're using this style...

import httpx

url = "http://example.com"
with httpx.Client(transport=httpx.HTTPTransport()) as client:
    print(client.get(url))

Rather than this style...

import httpx

url = "http://example.com"
with httpx.Client() as client:
    print(client.get(url))

[MarkWine]

Sorry to pop this back up, but may have extra context on why the transport might be used in this way and why results might be surprising. I think based on documentation and code examples, a new user might see the transport to be a simple 'plug in for extra features' addition to the client for something like retries. Looking at the requests library HTTPAdapter (which seems to be similar in most regards), environmental proxies are maintained, so in the following example:

import asyncio
import os

import httpx
from requests import Session
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

os.environ["https_proxy"] = "http://127.0.0.1:8080/"

async def test_transport_proxy():
    base_client = httpx.AsyncClient()
    transport_client = httpx.AsyncClient(transport=httpx.AsyncHTTPTransport(retries=2))
    await base_client.get("https://example.org/1")
    await transport_client.get("https://example.org/2")


def test_adapter_proxy():
    base_session = Session()
    transport_session = Session()
    transport_session.mount("https://", HTTPAdapter(max_retries=Retry(total=2)))
    base_session.get("https://example.org/3")
    transport_session.get("https://example.org/4")


asyncio.run(test_transport_proxy_a())
test_adapter_proxy()

The proxy is used in all but case 2.

I ran into this when we migrated an app to an environment that needed proxies for external traffic. Everything worked based on environment proxy settings except the one case in which we added retries when making a large number of calls to a somewhat flaky external service. It was straightforward to add proxy rules directly, so no complaints per se, but I was wondering on the rationale behind this and so thought I might jump in (also made similar comment in #1522 where this behavior was set).

[tscheburaschka]

I also tried to raise this issue in #2593 but was ignored at the time. I think the documentation could be more explicit for this case.