HTTP/3 proxy CONNECT and MASQUE CONNECT-UDP support (ngtcp2 QUIC) #21153
HTTP/3 proxy CONNECT and MASQUE CONNECT-UDP support (ngtcp2 QUIC) #21153aritrbas wants to merge 1 commit into
Conversation
|
I'm not sure why, but it seems git conflicts prevented the CI jobs from running (?) |
bagder
left a comment
There was a problem hiding this comment.
Because this is a huge change with lots of new pieces, I think we should declare that this functionality is experimental at first and make them opt-in at build time. This way we can let it in with less risk and we can let the functionality get tested and tweaked for a while first before we say it is production ready.
Some small gentle comments to start out with.
b99baae to
5b4d86f
Compare
bagder
left a comment
There was a problem hiding this comment.
1177:
curl_version_info.3: missing feature name PROXY-HTTP3
1275:
false positive: fix in #21191
1276:
run make optiontable in lib/ to get it updated as it likes
There are also some build errors in different build configurations, and C89 nits.
9dc79e1 to
0185fc8
Compare
0185fc8 to
8cb9fa3
Compare
There was a problem hiding this comment.
Pull request overview
This PR introduces experimental proxy support for HTTP/3 (QUIC) and MASQUE CONNECT-UDP when using the ngtcp2/nghttp3 HTTP/3 backend. It extends both libcurl and the curl CLI, adds a capsule-protocol implementation, and introduces new test infrastructure (h2o-based) plus unit/integration tests.
Changes:
- Add experimental proxy features: HTTP/3 proxy CONNECT (
--proxy-http3,CURLPROXY_HTTPS3) and MASQUE CONNECT-UDP (--proxyudptunnel,CURLOPT_HTTPPROXYUDPTUNNEL). - Implement capsule protocol helpers and extend proxy connection filters (H1/H2 + new H3 proxy filter), including QUIC filter-chain adjustments for UDP tunneling.
- Add unit and pytest integration coverage, including an h2o-based test proxy/server harness.
Reviewed changes
Copilot reviewed 69 out of 70 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
lib/cf-h1-proxy.c |
Adds CONNECT-UDP handling and capsule decapsulation in H1 proxy filter. |
lib/cf-h2-proxy.c |
Adds CONNECT-UDP request/response handling and capsule encapsulation/decapsulation in H2 proxy filter. |
lib/cf-h3-proxy.[ch] |
Introduces the new HTTP/3 proxy filter used when ALPN negotiates h3. |
lib/capsule.[ch] |
Implements capsule encoding/decoding helpers for HTTP Datagrams / CONNECT-UDP. |
lib/vquic/vquic.[ch], lib/vquic/curl_ngtcp2.[ch] |
Adjust QUIC send/recv paths to work over non-UDP lower filters (CONNECT-UDP tunnels). |
lib/connect.c, lib/url.c, lib/setopt.c, lib/version.c, include/curl/curl.h |
Wire up new proxy type/option, connection transport selection, and feature exposure. |
src/tool_getparam.[ch], src/tool_cfgable.h, src/tool_listhelp.c, src/config2setopts.c |
Add CLI flags/options and map them to libcurl options. |
tests/unit/unit3220.c, tests/data/test3220, tests/unit/Makefile.inc, tests/data/Makefile.am |
Add capsule protocol unit tests and integrate them into the unit test suite. |
tests/http/testenv/h2o.py, tests/http/testenv/env.py, tests/http/testenv/curl.py, tests/http/conftest.py |
Add h2o server/proxy harness, env detection, and proxy arg support for HTTP/3. |
tests/http/test_60_h3_proxy.py, tests/http/Makefile.am |
Add pytest integration suite for HTTP/3 proxy + CONNECT-UDP. |
configure.ac, CMakeLists.txt, lib/curl_config-cmake.h.in, docs/INSTALL-CMAKE.md, docs/EXPERIMENTAL.md, docs/tests/HTTP.md |
Add build-time opt-in for experimental HTTP/3 proxy support and document it. |
docs/cmdline-opts/*, docs/libcurl/opts/*, docs/libcurl/symbols-in-versions, docs/options-in-versions |
Document new CLI options and libcurl symbols/behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
017d85b to
56b39ff
Compare
|
There's still a merge conflict and some builds now fail a lot of tests. |
bf226e9 to
fdcf22c
Compare
- H3 proxy: re-sync code with original source `curl_ngtcp2.c` to reduce differences, and to apply missed minor fixes. Also apply clang-format. Drop redundant `#undef`s, casts, `#endif` comments, includes, drop intermediate variables, sync include and macro order. Follow-up to e78b1b3 #21153 - INSTALL-CMAKE.md: move `CURL_ENABLE_SMB` to the enable section. - tests/http/env: rename `tcpdmp` to `tcpdump` to match object variable. - mbedtls: drop incorrect `mbedTLS 4+` comments. (features are also supported by 3+, meaning it's always supported.) - lib1648: rename a variable to match purpose. - CIPHERS.md: alpha-sort link list. - replace rare `X''` hex markup with `0x`. - `IP v4/6` -> `IPv4/6`. - 'version X.Y' -> 'vX.Y', where sensible. - 'VX.Y' -> 'vX.Y', where sensible. - fix indents, casing, newlines, typos. Closes #21772
|
To test this, I tried enabling H3 proxy in CI. It builds fine, but pyests #21789 (comment) h2o doesn't seem to be packaged by Debian/Ubuntu. On macOS it's I wonder if there would be ways to fix the above issues, and/or perhaps |
Thanks for catching this! I have pushed fixes for these issues in #21791
Regarding h2o alternatives:Based on my CONNECT-UDP/MASQUE research:
I used h2o for these tests since it has full CONNECT-UDP support. |
|
Envoy isn’t going to help because it requires Bazel and uses a specific binary build of clang. |
Fix pytest failures in HTTP/3 proxy tests when h2o is not installed, misconfigured, or fails to start at runtime. This prevents: - FileNotFoundError when h2o document root does not exist - Fixture setup errors when h2o is configured but cannot start - Unused test data file creation when h2o is absent or broken - CI aborts on systems where h2o exists but is not runnable Bug: #21789 (comment) Bug: #21789 (comment) Follow-up to e78b1b3 #21153 Closes #21791
|
@aritrbas: One issue still lingers: with h2o not installed, h2o tests still want to run, and fail: https://github.com/curl/curl/actions/runs/26571899572/job/78280911172?pr=21789 |
Also: - GHA/http3-linux: enable deprecated APIs in openssl-prev local OpenSSL builds. Required by h2o and its vendored dependencies. Tried OpenSSL 4, LibreSSL 4.x, BoringSSL: all failed at one point. - GHA/http3-linux: build h2o from source. libuv1-dev may not be stricly required. Tried installing libwslay-dev, but it wasn't recognized. Also disable building h2o libs for a much smaller dist directory and slightly faster build. Sadly, h2o is not versioned, so I pinned to the current latest commit at the master branch. It advertises itself as 2.3.0-DEV in pytest. - drop redundant `libnghttp3` installs. Remains of openssl-quic builds. Follow-up to 6aaac9d #20226 Note GHA/macos pytests may or not not be stable with the H3 proxy tests. Follow-up to e78b1b3 #21153 Closes #21789
code: - less exception handling in existing code - true ip happy eyeballing - enable certificate verification - cf-h2-proxy: abort connection when server closed connection tests: - remove all --insecure and --proxy-insecure args - make session reuse test_60_12 a working one - resolve port conflicts between h2o and nghttpx - use proxy args better - make test_60_06 run shorter - kill h2o at the end of tests, normal stop takes too long Ref: 59213f8 #21789 Follow-up to e78b1b3 #21153 Closes #21798
- replace literal -1 with `NGHTTP3_ERR_CALLBACK_FAILURE` in nghttp3 callback. - replace `NGHTTP3_ERR_CALLBACK_FAILURE` with `NGTCP2_ERR_CALLBACK_FAILURE` in ngtcp2 callbacks. - test_60_h3_proxy: fix non-critical typo in symbol. Spotted by GitHub Code Quality Follow-up to e78b1b3 #21153 Closes #21802
- test_60_02_connect_tunnel_fail[fail_h3_over_h2_proxytunnel] - test_60_03_h3_target_auto_connect_udp[proxy_h3] - test_60_15_connect_timeout Follow-up to 59213f8 curl#21789 Follow-up to e78b1b3 curl#21153
- test_60_02_connect_tunnel_fail[fail_h1_over_h3_proxytunnel] - test_60_02_connect_tunnel_fail[fail_h3_over_h2_proxytunnel] - test_60_02_connect_tunnel_fail[fail_h3_over_h3_proxytunnel] - test_60_03_h3_target_auto_connect_udp[proxy_h3] - test_60_15_connect_timeout Further flaky ones may be disabled in future commits. All to be re-enabled after stabilizing them. Follow-up to 59213f8 #21789 Follow-up to e78b1b3 #21153 Closes #21803
- merge tests into a single class. For shorter names, to fix sort order by test number, and to align with other tests. - fix preconditions to make `test_60_04_guard_proxy_http3_unsupported` actually run. - replace local precondition with constant of the same effect. - drop redundant non-`ngtcp2` requirement for `test_60_04_guard_proxy_http3_unsupported`. (seemed relevant for no longer supported openssl-quic builds.) - drop unused `NGTCP2_ONLY_MSG` constant. Follow-up to e4139a7 #21798 - avoid creating unnecessary test data blobs, and minimize their scopes. Follow-up to 91facd7 #21791 Follow-up to e78b1b3 #21153 Closes #21811
This patch adds two major proxy capabilities to curl (ngtcp2 QUIC):
- HTTP/3 Proxy CONNECT: Tunnel HTTP/1.1 or HTTP/2 traffic through an
HTTPS proxy that speaks HTTP/3 (QUIC) using the standard CONNECT
method over an HTTP/3 connection.
- MASQUE CONNECT-UDP: Tunnel HTTP/3 (QUIC) traffic through an HTTP
proxy (speaking HTTP/1.1, HTTP/2, or HTTP/3) using the extended
CONNECT method with the CONNECT-UDP protocol (RFC9297 & RFC9298).
Public API additions:
- `CURLPROXY_HTTPS3`: new proxy type constant for HTTP/3 proxy
- `--proxy-http3`: new CLI flag to negotiate HTTP/3 with HTTPS proxy
The implementation adds two new filters:
- `H3-PROXY` - enables negotiating HTTP/3 (QUIC) to the proxy and
running CONNECT/CONNECT-UDP through that proxy transport.
- `CAPSULE` - dedicated filter inserted between QUIC transport and
HTTP-PROXY to handle datagram capsule encapsulation/decapsulation.
Here is how the curl filter chaining looks in different scenarios:
- HTTP/3 Proxy CONNECT (tunneling TCP protocols over QUIC proxy):
conn -> HTTP/1.1 or HTTP/2 -> SSL -> HTTP-PROXY ->
H3-PROXY -> HAPPY-EYEBALLS -> UDP
- MASQUE CONNECT-UDP (tunneling QUIC over any proxy):
conn -> HTTP/3 -> CAPSULE -> HTTP-PROXY -> H3-PROXY ->
HAPPY-EYEBALLS -> UDP
conn -> HTTP/3 -> CAPSULE -> HTTP-PROXY -> H1-PROXY or H2-PROXY ->
SSL -> HAPPY-EYEBALLS -> TCP
- Both features currently require the ngtcp2 QUIC backend.
- Both features are experimental (disabled by default). Enable with
`--enable-proxy-http3`(autotools) or `-DUSE_PROXY_HTTP3=ON`(CMake).
Tests:
- tests/unit/unit3400.c: Unit tests for capsule protocol encode/decode
- tests/http/test_60_h3_proxy.py: Comprehensive pytest integration suite
- tests/http/testenv/h2o.py: Managing h2o instances with HTTP/1.1, HTTP/2,
and HTTP/3 (QUIC) listeners, proxy.connect and proxy.connect-udp enabled.
References:
RFC 9297 - HTTP Datagrams and the Capsule Protocol
RFC 9298 - Proxying UDP in HTTP
RFC 9000 §16 — Variable-Length Integer Encoding
Signed-off-by: Aritra Basu <aritrbas+gh@cisco.com>
Closes curl#21153
- H3 proxy: re-sync code with original source `curl_ngtcp2.c` to reduce differences, and to apply missed minor fixes. Also apply clang-format. Drop redundant `#undef`s, casts, `#endif` comments, includes, drop intermediate variables, sync include and macro order. Follow-up to e78b1b3 curl#21153 - INSTALL-CMAKE.md: move `CURL_ENABLE_SMB` to the enable section. - tests/http/env: rename `tcpdmp` to `tcpdump` to match object variable. - mbedtls: drop incorrect `mbedTLS 4+` comments. (features are also supported by 3+, meaning it's always supported.) - lib1648: rename a variable to match purpose. - CIPHERS.md: alpha-sort link list. - replace rare `X''` hex markup with `0x`. - `IP v4/6` -> `IPv4/6`. - 'version X.Y' -> 'vX.Y', where sensible. - 'VX.Y' -> 'vX.Y', where sensible. - fix indents, casing, newlines, typos. Closes curl#21772
This PR adds two major proxy capabilities to curl, targeting the ngtcp2 QUIC backend:
Public API additions:
CURLPROXY_HTTPS3: new proxy type constant for HTTP/3 proxy--proxy-http3: new CLI flag to negotiate HTTP/3 with an HTTPS proxyThe implementation adds two new filters:
H3-PROXY- enables negotiating HTTP/3 (QUIC) to the proxy and running CONNECT/CONNECT-UDP through that proxy transport.CAPSULE- dedicated filter inserted between QUIC transport andHTTP-PROXYto handle datagram capsule encapsulation/decapsulation.Here is how the curl filter chaining looks in different scenarios:
Usage:
HTTP/3 Proxy CONNECT (
--proxy-http3) and MASQUE CONNECT-UDP (--proxyudptunnel) are experimental (disabled by default). Enable with--enable-proxy-http3(autotools) or-DUSE_PROXY_HTTP3=ON(CMake).Tests:
H2oServer(QUIC-enabled file server) and (ii)H2oProxy(MASQUE proxy with HTTP/1.1 + HTTP/2 + HTTP/3 listeners, proxy.connect and proxy.connect-udp enabled) for testing.References: