Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the 'proxy' parameter and deprecate 'proxies'. #2879

Merged
merged 23 commits into from Dec 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
170cbef
Add the proxy parameter and deprecate proxies
karpetrosyan Oct 6, 2023
189a4f2
Merge branch 'master' into drop-proxies
tomchristie Oct 10, 2023
88eb99b
Make the Client.proxy and HTTPTransport.proxy types the same
karpetrosyan Oct 10, 2023
11bec08
Update httpx/_transports/default.py
tomchristie Oct 11, 2023
fad66cf
Update httpx/_transports/default.py
tomchristie Oct 11, 2023
d16f9f3
Drop unneeded noqa
karpetrosyan Oct 11, 2023
64a4596
Changelog
karpetrosyan Oct 11, 2023
541cc31
update documentation
karpetrosyan Oct 12, 2023
812a70b
Allow None in mounts
karpetrosyan Oct 12, 2023
ccf1803
typos
karpetrosyan Oct 12, 2023
bacd8b2
Update httpx/_types.py
karpetrosyan Oct 12, 2023
d5fa2b0
Changes proxies to proxy in CLI app
karpetrosyan Oct 12, 2023
17c6cde
Add proxy to request function
karpetrosyan Oct 12, 2023
5aed92e
Merge branch 'drop-proxies' of github.com:karosis88/httpx into drop-p…
karpetrosyan Oct 12, 2023
6e5e4ac
Merge branch 'master' into drop-proxies
karpetrosyan Oct 29, 2023
2672d8a
Merge branch 'master' into drop-proxies
tomchristie Nov 2, 2023
fffa454
Merge branch 'master' into drop-proxies
karpetrosyan Nov 17, 2023
4deebb8
Merge branch 'master' into drop-proxies
karpetrosyan Nov 17, 2023
0b73817
Update CHANGELOG.md
karpetrosyan Dec 11, 2023
1e3bf1e
Update docs/troubleshooting.md
karpetrosyan Dec 11, 2023
725a32e
Update docs/troubleshooting.md
karpetrosyan Dec 11, 2023
cbd63da
Merge branch 'master' into drop-proxies
karpetrosyan Dec 11, 2023
c7971ce
Lint
karpetrosyan Dec 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Expand Up @@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

## Unreleased

### Changed

* The `proxies` argument is now deprecated. You should use the `proxy` argument instead, or use `mounts=` for more complex configurations. (#2879)

### Fixed

* Allow URLs where username or password contains unescaped '@'. (#2986)
Expand Down
264 changes: 132 additions & 132 deletions docs/advanced.md
Expand Up @@ -504,7 +504,7 @@ The `NetRCAuth()` class uses [the `netrc.netrc()` function from the Python stand

## HTTP Proxying

HTTPX supports setting up [HTTP proxies](https://en.wikipedia.org/wiki/Proxy_server#Web_proxy_servers) via the `proxies` parameter to be passed on client initialization or top-level API functions like `httpx.get(..., proxies=...)`.
HTTPX supports setting up [HTTP proxies](https://en.wikipedia.org/wiki/Proxy_server#Web_proxy_servers) via the `proxy` parameter to be passed on client initialization or top-level API functions like `httpx.get(..., proxy=...)`.

<div align="center">
<img src="https://upload.wikimedia.org/wikipedia/commons/thumb/2/27/Open_proxy_h2g2bob.svg/480px-Open_proxy_h2g2bob.svg.png"/>
Expand All @@ -516,19 +516,19 @@ HTTPX supports setting up [HTTP proxies](https://en.wikipedia.org/wiki/Proxy_ser
To route all traffic (HTTP and HTTPS) to a proxy located at `http://localhost:8030`, pass the proxy URL to the client...

```python
with httpx.Client(proxies="http://localhost:8030") as client:
with httpx.Client(proxy="http://localhost:8030") as client:
...
```

For more advanced use cases, pass a proxies `dict`. For example, to route HTTP and HTTPS requests to 2 different proxies, respectively located at `http://localhost:8030`, and `http://localhost:8031`, pass a `dict` of proxy URLs:
For more advanced use cases, pass a mounts `dict`. For example, to route HTTP and HTTPS requests to 2 different proxies, respectively located at `http://localhost:8030`, and `http://localhost:8031`, pass a `dict` of proxy URLs:

```python
proxies = {
"http://": "http://localhost:8030",
"https://": "http://localhost:8031",
proxy_mounts = {
"http://": httpx.HTTPTransport(proxy="http://localhost:8030"),
"https://": httpx.HTTPTransport(proxy="http://localhost:8031"),
}

with httpx.Client(proxies=proxies) as client:
with httpx.Client(mounts=proxy_mounts) as client:
...
```

Expand All @@ -546,132 +546,10 @@ For detailed information about proxy routing, see the [Routing](#routing) sectio
Proxy credentials can be passed as the `userinfo` section of the proxy URL. For example:

```python
proxies = {
"http://": "http://username:password@localhost:8030",
# ...
}
```

### Routing

HTTPX provides fine-grained controls for deciding which requests should go through a proxy, and which shouldn't. This process is known as proxy routing.

The `proxies` dictionary maps URL patterns ("proxy keys") to proxy URLs. HTTPX matches requested URLs against proxy keys to decide which proxy should be used, if any. Matching is done from most specific proxy keys (e.g. `https://<domain>:<port>`) to least specific ones (e.g. `https://`).

HTTPX supports routing proxies based on **scheme**, **domain**, **port**, or a combination of these.

#### Wildcard routing

Route everything through a proxy...

```python
proxies = {
"all://": "http://localhost:8030",
}
```

#### Scheme routing

Route HTTP requests through one proxy, and HTTPS requests through another...

```python
proxies = {
"http://": "http://localhost:8030",
"https://": "http://localhost:8031",
}
```

#### Domain routing

Proxy all requests on domain "example.com", let other requests pass through...

```python
proxies = {
"all://example.com": "http://localhost:8030",
}
```

Proxy HTTP requests on domain "example.com", let HTTPS and other requests pass through...

```python
proxies = {
"http://example.com": "http://localhost:8030",
}
```

Proxy all requests to "example.com" and its subdomains, let other requests pass through...

```python
proxies = {
"all://*example.com": "http://localhost:8030",
}
```

Proxy all requests to strict subdomains of "example.com", let "example.com" and other requests pass through...

```python
proxies = {
"all://*.example.com": "http://localhost:8030",
}
```

#### Port routing

Proxy HTTPS requests on port 1234 to "example.com"...

```python
proxies = {
"https://example.com:1234": "http://localhost:8030",
}
```

Proxy all requests on port 1234...

```python
proxies = {
"all://*:1234": "http://localhost:8030",
}
```

#### No-proxy support

It is also possible to define requests that _shouldn't_ be routed through proxies.

To do so, pass `None` as the proxy URL. For example...

```python
proxies = {
# Route requests through a proxy by default...
"all://": "http://localhost:8031",
# Except those for "example.com".
"all://example.com": None,
}
```

#### Complex configuration example

You can combine the routing features outlined above to build complex proxy routing configurations. For example...

```python
proxies = {
# Route all traffic through a proxy by default...
"all://": "http://localhost:8030",
# But don't use proxies for HTTPS requests to "domain.io"...
"https://domain.io": None,
# And use another proxy for requests to "example.com" and its subdomains...
"all://*example.com": "http://localhost:8031",
# And yet another proxy if HTTP is used,
# and the "internal" subdomain on port 5550 is requested...
"http://internal.example.com:5550": "http://localhost:8032",
}
with httpx.Client(proxy="http://username:password@localhost:8030") as client:
...
```

#### Environment variables

HTTP proxying can also be configured through environment variables, although with less fine-grained control.

See documentation on [`HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`](environment_variables.md#http_proxy-https_proxy-all_proxy) for more information.

### Proxy mechanisms

!!! note
Expand Down Expand Up @@ -707,7 +585,7 @@ $ pip install httpx[socks]
You can now configure a client to make requests via a proxy using the SOCKS protocol:

```python
httpx.Client(proxies='socks5://user:pass@host:port')
httpx.Client(proxy='socks5://user:pass@host:port')
```

## Timeout Configuration
Expand Down Expand Up @@ -1294,3 +1172,125 @@ Adding support for custom schemes:
mounts = {"file://": FileSystemTransport()}
client = httpx.Client(mounts=mounts)
```

### Routing

HTTPX provides a powerful mechanism for routing requests, allowing you to write complex rules that specify which transport should be used for each request.

The `mounts` dictionary maps URL patterns to HTTP transports. HTTPX matches requested URLs against URL patterns to decide which transport should be used, if any. Matching is done from most specific URL patterns (e.g. `https://<domain>:<port>`) to least specific ones (e.g. `https://`).

HTTPX supports routing requests based on **scheme**, **domain**, **port**, or a combination of these.

#### Wildcard routing

Route everything through a transport...

```python
mounts = {
"all://": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

#### Scheme routing

Route HTTP requests through one transport, and HTTPS requests through another...

```python
mounts = {
"http://": httpx.HTTPTransport(proxy="http://localhost:8030"),
"https://": httpx.HTTPTransport(proxy="http://localhost:8031"),
}
```

#### Domain routing

Proxy all requests on domain "example.com", let other requests pass through...

```python
mounts = {
"all://example.com": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

Proxy HTTP requests on domain "example.com", let HTTPS and other requests pass through...

```python
mounts = {
"http://example.com": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

Proxy all requests to "example.com" and its subdomains, let other requests pass through...

```python
mounts = {
"all://*example.com": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

Proxy all requests to strict subdomains of "example.com", let "example.com" and other requests pass through...

```python
mounts = {
"all://*.example.com": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

#### Port routing

Proxy HTTPS requests on port 1234 to "example.com"...

```python
mounts = {
"https://example.com:1234": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

Proxy all requests on port 1234...

```python
mounts = {
"all://*:1234": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

#### No-proxy support

It is also possible to define requests that _shouldn't_ be routed through the transport.

To do so, pass `None` as the proxy URL. For example...

```python
mounts = {
# Route requests through a proxy by default...
"all://": httpx.HTTPTransport(proxy="http://localhost:8031"),
# Except those for "example.com".
"all://example.com": None,
}
```

#### Complex configuration example

You can combine the routing features outlined above to build complex proxy routing configurations. For example...

```python
mounts = {
# Route all traffic through a proxy by default...
"all://": httpx.HTTPTransport(proxy="http://localhost:8030"),
# But don't use proxies for HTTPS requests to "domain.io"...
"https://domain.io": None,
# And use another proxy for requests to "example.com" and its subdomains...
"all://*example.com": httpx.HTTPTransport(proxy="http://localhost:8031"),
# And yet another proxy if HTTP is used,
# and the "internal" subdomain on port 5550 is requested...
"http://internal.example.com:5550": httpx.HTTPTransport(proxy="http://localhost:8032"),
}
```

#### Environment variables

There are also environment variables that can be used to control the dictionary of the client mounts.
They can be used to configure HTTP proxying for clients.

See documentation on [`HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`](environment_variables.md#http_proxy-https_proxy-all_proxy) for more information.

12 changes: 8 additions & 4 deletions docs/compatibility.md
Expand Up @@ -157,13 +157,17 @@ httpx.get('https://www.example.com', timeout=None)

## Proxy keys

When using `httpx.Client(proxies={...})` to map to a selection of different proxies, we use full URL schemes, such as `proxies={"http://": ..., "https://": ...}`.
HTTPX uses the mounts argument for HTTP proxying and transport routing.
It can do much more than proxies and allows you to configure more than just the proxy route.
For more detailed documentation, see [Mounting Transports](advanced.md#mounting-transports).

When using `httpx.Client(mounts={...})` to map to a selection of different transports, we use full URL schemes, such as `mounts={"http://": ..., "https://": ...}`.

This is different to the `requests` usage of `proxies={"http": ..., "https": ...}`.

This change is for better consistency with more complex mappings, that might also include domain names, such as `proxies={"all://": ..., "all://www.example.com": None}` which maps all requests onto a proxy, except for requests to "www.example.com" which have an explicit exclusion.
This change is for better consistency with more complex mappings, that might also include domain names, such as `mounts={"all://": ..., httpx.HTTPTransport(proxy="all://www.example.com": None})` which maps all requests onto a proxy, except for requests to "www.example.com" which have an explicit exclusion.

Also note that `requests.Session.request(...)` allows a `proxies=...` parameter, whereas `httpx.Client.request(...)` does not.
Also note that `requests.Session.request(...)` allows a `proxies=...` parameter, whereas `httpx.Client.request(...)` does not allow `mounts=...`.

## SSL configuration

Expand Down Expand Up @@ -195,7 +199,7 @@ We don't support `response.is_ok` since the naming is ambiguous there, and might

There is no notion of [prepared requests](https://requests.readthedocs.io/en/stable/user/advanced/#prepared-requests) in HTTPX. If you need to customize request instantiation, see [Request instances](advanced.md#request-instances).

Besides, `httpx.Request()` does not support the `auth`, `timeout`, `follow_redirects`, `proxies`, `verify` and `cert` parameters. However these are available in `httpx.request`, `httpx.get`, `httpx.post` etc., as well as on [`Client` instances](advanced.md#client-instances).
Besides, `httpx.Request()` does not support the `auth`, `timeout`, `follow_redirects`, `mounts`, `verify` and `cert` parameters. However these are available in `httpx.request`, `httpx.get`, `httpx.post` etc., as well as on [`Client` instances](advanced.md#client-instances).

## Mocking

Expand Down
4 changes: 1 addition & 3 deletions docs/contributing.md
Expand Up @@ -213,9 +213,7 @@ this is where our previously generated `client.pem` comes in:
```
import httpx

proxies = {"all://": "http://127.0.0.1:8080/"}

with httpx.Client(proxies=proxies, verify="/path/to/client.pem") as client:
with httpx.Client(proxy="http://127.0.0.1:8080/", verify="/path/to/client.pem") as client:
response = client.get("https://example.org")
print(response.status_code) # should print 200
```
Expand Down
16 changes: 9 additions & 7 deletions docs/troubleshooting.md
Expand Up @@ -19,9 +19,9 @@ httpx.ProxyError: _ssl.c:1091: The handshake operation timed out
**Resolution**: it is likely that you've set up your proxies like this...

```python
proxies = {
"http://": "http://myproxy.org",
"https://": "https://myproxy.org",
mounts = {
"http://": httpx.HTTPTransport(proxy="http://myproxy.org"),
"https://": httpx.HTTPTransport(proxy="https://myproxy.org"),
}
```

Expand All @@ -32,16 +32,18 @@ But if you get the error above, it is likely that your proxy doesn't support con
Change the scheme of your HTTPS proxy to `http://...` instead of `https://...`:

```python
proxies = {
"http://": "http://myproxy.org",
"https://": "http://myproxy.org",
mounts = {
"http://": httpx.HTTPTransport(proxy="http://myproxy.org"),
"https://": httpx.HTTPTransport(proxy="http://myproxy.org"),
}
```

This can be simplified to:

```python
proxies = "http://myproxy.org"
proxy = "http://myproxy.org"
with httpx.Client(proxy=proxy) as client:
...
```

For more information, see [Proxies: FORWARD vs TUNNEL](advanced.md#forward-vs-tunnel).
Expand Down