Proposal 4.0 (HTTP/2+3, OS Trust Store, Custom DNS, OCSP, ...)

[Ousret]

This PR showcases how HTTPie could evolve outside of Requests.

Niquests is supposed to be a (mostly) compatible fork of Requests.

Try this preview:

$ pip install "git+https://github.com/Ousret/HTTPie.git@feature-tryout-niquests" -U

Here are the biggest pros of this:

• OS truststore by default, no more certifi!
• Object-oriented headers. Could bring additional features!
• Fully type-annotated!
• HTTP/3 over QUIC.
• HTTP/2 by default.
• Exit Python http.client! in favor of h11.
• Timeout by default.
• Inspect peer certificate, HTTP version, TLS version, cipher, and so on via hook/callback before a request is sent.
• All the features you expose are available in all three protocols.
• Python 3.7+, no sacrifice needed.
• Encrypted DNS support. w/ DNS-over-HTTPS, DNS-over-QUIC, DNS-over-TLS and plain DNS-over-UDP.

Obviously, cons:

• Stricter on emitted requests per RFCs
• Young project but based on solid bases, knowledge, and experiences.
• Need to publish new packages to distro. Easy but time-consuming
• Exit pyopenssl, not supported. more of a pro to me
• Require major bump? Could be. Should be.

Complete list of changes in the fork: https://github.com/jawah/niquests/blob/main/HISTORY.md

• Close httpie refuses to trust a self-signed certificate #576
• Close HTTPie ignores system certificates #480
• Close Support HTTP/2 (httpie-http2 plugin doesn't work) #692
• Close enable quic protocol #523
• Close show me the SSL certificate the server is using? #549
• Close System for meta information output options (timing, SSL info, …) #1023
• Close Include SSL/TLS and other metadata information in --meta/-vv #826
• Close "SSL: CERTIFICATE_VERIFY_FAILED" should show cert at least with --debug #800
• Close Display used server IP in verbose mode #1495
• Close SSL/TLS server certificate validation #1202
• Close Add equivalent cURL --resolve flag #99
• Close binding to an IP #1422
• Close Add -4 and -6 switches to force IP protocol version #94
• Close I would like the option to disable the DNS Cache and do name resolution on every request #1480
• Close Unable to install httpie via Chocolatey #1522
• Close Multiple response headers with same name combined to a single comma-separated list #1413
• Close Digest authentication with qop=auth-int fails "TypeError: expected string or bytes-like object"  #1446
• Close Should use the user-provided Host header for SNI #414
• Close HTTPie could be lead to believe data was passed in stdin when it was not #1551
• Close add local-port flag #1456
• Close can't install chocolatey issues with python error code 1603 #1338
• Close Bad filename when having non ASCII characters #1401
• Close Request is fired to packages.httpie.io on every invocation of http #1527
• Close Cookies with "Domain=localhost" aren't getting stored in session file #602
• Close Change insecure SSL/TLS to deprecated #962
• Close httpie -d does not work with gzip compressed content #1554
• Close --download reports an "incomplete download" with Content-Encoding: gzip and content-length. #423
• Close Deprecate PROTOCOL_SSLv23 #1400
• Close Feature Request: Support special-use domain names like localhost. #1458
• Close --ssl — TLS 1.3 & Python 3.7 compatibility  #722
• Close httpie - support for manipulating hostheaders? #579
• Close Installation using chocolatey fails installing to system Python environment #1580
• Close Unable to access HTTPS URLs on Windows securely #1581

---

4.0.0.b1 (unreleased)

• Make it possible to unset the User-Agent, and Accept-Encoding headers. (#1502)
• Dependency on requests was changed in favor of compatible niquests. (#1531)
• Added support for HTTP/2, and HTTP/3 protocols. (#523) (#692) (#1531)
• Added request metadata for the TLS certificate, negotiated version with cipher, the revocation status and the remote peer IP address. (#1495) (#1023) (#826) (#1531)
• Added support to load the operating system trust store for the peer certificate validation. (#480) (#1531)
• Added detailed timings in response metadata with DNS resolution, established, TLS handshake, and request sending delays. (#1023) (#1531)
• Added support for using alternative DNS resolver using --resolver. DNS over HTTPS, DNS over TLS, DNS over QUIC, and DNS over UDP are accepted. (#99) (#1531)
• Added support for binding to a specific network adapter with --interface. (#1422) (#1531)
• Added support for specifying the local port with --local-port. (#1456) (#1531)
• Added support for forcing either IPv4 or IPv6 to reach the remote HTTP server with -6 or -4. (#94) (#1531)
• Removed support for pyopenssl. (#1531)
• Removed support for dead SSL protocols < TLS 1.0 (e.g. sslv3) as per pyopenssl removal. (#1531)
• Dropped dependency on requests_toolbelt in favor of directly including MultipartEncoder into HTTPie due to its direct dependency to requests. (#1531)
• Dropped dependency on multidict in favor of implementing an internal one due to often missing pre-built wheels. (#1522) (#1531)
• Fixed the case when multiple headers where concatenated in the response output. (#1413) (#1531)
• Fixed an edge case where HTTPie could be lead to believe data was passed in stdin, thus sending a POST by default. (#1551) (#1531)
  This fix has the particularity to consider 0 byte long stdin buffer as absent stdin. Empty stdin buffer will be ignored.
• Slightly improved performance while downloading by setting chunk size to -1 to retrieve packets as they arrive. (#1531)
• Added support for using the system trust store to retrieve root CAs for verifying TLS certificates. (#1531)
• Removed support for keeping the original casing of HTTP headers. This come from an outer constraint by newer protocols, namely HTTP/2+ that normalize header keys by default.
  From the HTTPie user perspective, they are "prettified" on the output by default. e.g. "x-hello-world" is displayed as "X-Hello-World".
• Fixed multipart form data having filename not rfc2231 compliant when name contain non-ascii characters. (#1401)
• Fixed issue where the configuration directory was not created at runtime that made the update fetcher run everytime. (#1527)
• Fixed cookie persistence in HTTPie session when targeting localhost. They were dropped due to the standard library. (#1527)
• Fixed downloader when trying to fetch compressed content. The process will no longer exit with the "Incomplete download" error. (#1554) (#423) (#1527)
• Added automated resolution of hosts ending with .localhost to the default loopback address. (#1458) (#1527)

Existing plugins are expected to work without any changes. The only caveat would be that certain plugin explicitly require requests.
Future contributions may be made in order to relax the constraints where applicable.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (4d7d6b6) to head (7cd6579).
Report is 370 commits behind head on master.

❗ Current head 7cd6579 differs from pull request most recent head 55aea9c

Please upload reports for the commit 55aea9c to get more accurate results.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #1531       +/-   ##
==========================================
- Coverage   97.28%       0   -97.29%     
==========================================
  Files          67       0       -67     
  Lines        4235       0     -4235     
==========================================
- Hits         4120       0     -4120     
+ Misses        115       0      -115     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Ousret]

So far, I identified this:

• Close httpie refuses to trust a self-signed certificate #576
• Close HTTPie ignores system certificates #480
• Close Support HTTP/2 (httpie-http2 plugin doesn't work) #692
• Close enable quic protocol #523
• Close show me the SSL certificate the server is using? #549

[Ousret]

We can add the following arguments and document them:

• --disable-http2
• --disable-http3
• --http3, e.g. force trying it.

[Ousret]

I've added and documented it.

• --disable-http2
• --disable-http3
• --http3, e.g. force trying it.

Ideally, the connection information (TLS, Cipher, Peer/Issuer Certificate, and OCSP) should be in the request metadata and with a lexer instruction.

[Ousret]

Now, the result is pleasant to the eye.
I have implemented the metadata for Request and the result is as follows:

Niquests now support tracking upload progress, this can also be done here. It should be straightforward enough.

[dwt]

Damn this looks so tantalizing. I'd love for this to get packaged as a tryout / beta. Something installable via e.g. pip install httpie[niquest] perhaps?

[janbrasna]

docs typos:

[janbrasna]

A couple of questions regarding how to surface some of the functionality and constraints to the users:

[Ousret]

In the latest commit:

• Added a way to force http2, http1
• So, now, we can enforce http2 by disabling http1
• http2 with prior knowledge is supported
• fixed the "real" speed download measured and reported when remote compressed the body
• disabled recently added test cases that tried to simulate gzip/br encoded bodies when the underlying library "responses" isn't really cut for it.