Add support for zstd decoding #3139
Conversation
mbeijen
force-pushed
the
zstd
branch
2 times, most recently
from
835d74a
to
01b4c6d
Compare
mbeijen
force-pushed
the
zstd
branch
3 times, most recently
from
09b31a1
to
4860886
Compare
mbeijen
force-pushed
the
zstd
branch
2 times, most recently
from
c531b62
to
4f2ed51
Compare
|
This is a lovely bit of functionality. Missing some coverage at the moment. |
Thanks. I will see if I can add tests for full coverage |
mbeijen
force-pushed
the
zstd
branch
2 times, most recently
from
10f21ba
to
96bde3f
Compare
|
I've added full test coverage now, again by borrowing a test inspired from urllib3 |
Sure thing. (thanks)
|
I've added a code comment in the appropriate places. In the docs and README there is already a credit given to urllib3 as a source for much of the lower level details, this is not much different. It is fair to add some extra credit here as well as a pointer for a reader to check the urllib3 implementatition in case of changes.
Sure! Support for zstd is not really common on web sites yet. libcurl has support for some years now. Chrome has it now enabled by default so I will assume this might help? Also, Mozilla changed its stance on zstd from 'defer' to 'positive' in October, ref: mozilla/standards-positions#775 (comment) -- it is particularly well-suited for compressing JSON data. Sites that support it include instagram.com, facebook.com, caddyserver.com
I've added an entry. |
This adds support for zstd decoding using the python package zstandard. This is similar to how it is implemented in urllib3. I also chose the optional installation option httpx[zstd] to mimic the same option in urllib3. zstd decoding is similar to brotli, but in benchmarks it is supposed to be even faster. The zstd compression is described in RFC 8878. See encode#1986 Co-authored-by: Kamil Monicz <kamil@monicz.dev>
There was a problem hiding this comment.
Really nice work
$ httpx -v https://caddyserver.com
* Connecting to 'caddyserver.com'
* Connected to '165.227.20.207' on port 443
* SSL established using 'TLSv1.3' / 'TLS_AES_128_GCM_SHA256'
* Selected ALPN protocol: 'http/1.1'
* Server certificate:
* subject:
* commonName: 'caddyserver.com'
* issuer:
* countryName: 'US'
* organizationName: "Let's Encrypt"
* commonName: 'R3'
* version: 3
* serialNumber: '0334F48787F76D10C8FF7D436F4AB8C56B0F'
* notBefore: 'Feb 26 23:44:40 2024 GMT'
* notAfter: 'May 26 23:44:39 2024 GMT'
* subjectAltName:
* DNS: 'caddyserver.com'
* OCSP:
* 'http://r3.o.lencr.org'
* caIssuers:
* 'http://r3.i.lencr.org/'
GET / HTTP/1.1
Host: caddyserver.com
Accept: */*
Accept-Encoding: gzip, deflate, br, zstd
Connection: keep-alive
User-Agent: python-httpx/0.27.0
HTTP/1.1 200 OK
Alt-Svc: h3=":443"; ma=2592000
Content-Encoding: zstd
Content-Type: text/html; charset=utf-8
Server: Caddy
Vary: Accept-Encoding
Date: Thu, 21 Mar 2024 10:12:38 GMT
Transfer-Encoding: chunked
<!DOCTYPE html>
<html>
<head>
<title>Caddy - The Ultimate Server with Automatic HTTPS</title>
This adds support for zstd decoding using the python package zstandard. This is similar to how it is implemented in urllib3. I also chose the optional installation option httpx[zstd] to mimic the same option in urllib3. zstd decoding is similar to brotli, but in benchmarks it is supposed to be even faster. The zstd compression is described in RFC 8878. See encode#1986 Co-authored-by: Kamil Monicz <kamil@monicz.dev>
Summary
This adds support for zstd decoding using the python package zstandard. This is similar to how it is implemented in urllib3. I also chose the optional installation option httpx[zstd] to mimic the same option in urllib3.
zstd decoding is similar to brotli, but in benchmarks it is supposed to be even faster. The zstd compression is described in RFC 8878.
See #1986
Checklist