Add QMux support (HTTP/3 over TLS/TCP) by maskit · Pull Request #13465 · apache/trafficserver

maskit · 2026-07-31T16:19:53Z

HTTP/3 requires UDP, which is blocked or degraded on many networks. QMux carries QUIC stream multiplexing over a TLS/TCP connection so HTTP/3 can be served where UDP is unavailable. Server side only.

The transport is abstracted behind the existing QUICConnection and QUICStreamIO interfaces, so HTTP/3 session and application handling is reused unchanged. QMux is offered via ALPN "h3qx-01" on TLS ports.

The two transports are now selected independently of the QUIC backend. ENABLE_QUIC carries QUIC over UDP and defaults to on whenever a backend is available; ENABLE_QMUX carries it over TLS/TCP and defaults to on whenever quiche is built with qmux support. Either can be enabled without the other.

Copilot

Pull request overview

This PR adds server-side QMux (QUIC multiplexing over TLS/TCP) as an additional transport for HTTP/3, allowing HTTP/3 semantics to be served when UDP QUIC is unavailable. It integrates QMux behind existing QUICConnection / QUICStreamIO abstractions, adds a new ALPN (h3qx-01), and introduces a new build option to enable/disable QMux independently of UDP QUIC.

Changes:

Introduces h3qx-01 as a protocol tag / ALPN constant and wires it through the protocol registry.
Adds a new QMuxConnection implementation (quiche qmux-enabled) and integrates it with TLS accept / HTTP/3 accept.
Updates CMake to separate QUIC backend availability from transport enablement (TS_USE_QUIC vs TS_USE_QMUX) and links/builds the new ts::qmux library.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
CMakeLists.txt	Adds transport-level options and backend/transport selection logic for QUIC vs QMux.
include/iocore/net/qmux/QMuxConnection.h	Adds the public interface for the new QMux-backed `QUICConnection`.
include/iocore/net/quic/QUICStream.h	Makes `QUICStream` destructor virtual (polymorphic safety).
include/ts/apidefs.h.in	Exposes new ALPN / protocol tag constants and indices for `h3qx-01`.
include/tscore/ink_config.h.cmake.in	Adds `TS_USE_QMUX` compile-time define.
include/tscore/ink_inet.h	Declares `IP_PROTO_TAG_H3QX`.
src/iocore/net/CMakeLists.txt	Builds/links QUIC and QMux components based on transport flags; adds qmux subdir.
src/iocore/net/P_SSLNetVConnection.h	Adds conditional `QUICSupport` and stores a `QMuxConnection` on TLS VCs when enabled.
src/iocore/net/SSLNetVConnection.cc	Creates and clears `QMuxConnection` based on negotiated ALPN `h3qx-01`.
src/iocore/net/qmux/CMakeLists.txt	Introduces `qmux` static library target and alias `ts::qmux`.
src/iocore/net/qmux/QMuxConnection.cc	Implements QMux I/O bridging and QUIC stream operations on top of TLS/TCP.
src/proxy/CMakeLists.txt	Builds http3 subdirectory when either QUIC transport is enabled.
src/proxy/http/CMakeLists.txt	Links http3 support when either QUIC transport is enabled.
src/proxy/http/HttpProxyServerMain.cc	Registers `h3qx-01` ALPN endpoint on TLS ports when QMux is enabled.
src/proxy/http3/CMakeLists.txt	Links http3 against `ts::qmux` when QMux is enabled.
src/proxy/http3/Http3SessionAccept.cc	Accepts `h3qx-01` and wires QMux startup alongside HTTP/3 app startup.
src/records/RecHttp.cc	Adds `h3qx-01` to ALPN constants, indices, and OpenSSL wire-format conversion.
src/traffic_server/CMakeLists.txt	Links traffic_server against http3/quic libs when QUIC or QMux is enabled; adds qmux link.
src/traffic_server/traffic_server.cc	Initializes HTTP/3 config / subsystem when QUIC or QMux is enabled.
src/tscore/ink_inet.cc	Defines the new protocol tag constant for `h3qx-01`.

Copilot

Pull request overview

Copilot reviewed 26 out of 27 changed files in this pull request and generated 4 comments.

Suppressed comments (3)

src/iocore/net/qmux/QMuxConnection.cc:184

EVENT_INTERVAL drives quiche_conn_on_timeout(), but this connection never schedules interval/timeout events (unlike QUICNetVConnection, which schedules based on quiche_conn_timeout_as_millis). As a result, quiche’s idle timeout (and any other timer-driven state) won’t fire, which can leave hung QMux sessions consuming resources indefinitely.

  case EVENT_INTERVAL:
    quiche_conn_on_timeout(_quiche_con);
    _flush_quiche_output();
    break;

src/proxy/http3/Http3SessionAccept.cc:96

If ALPN negotiates h3qx-01 but qc is not actually a QMuxConnection, the code silently skips start() and continues, which will leave the connection stalled with no I/O loop driving quiche. Treat this as an error and close the netvc instead of continuing.

      auto *qmux_con = dynamic_cast<QMuxConnection *>(qc);
      if (qmux_con) {
        qmux_con->start(netvc);
      }

src/iocore/net/SSLNetVConnection.cc:1496

QMuxConnection construction can fail internally (it sets itself closed when _quiche_con can’t be created), but the handshake path doesn’t check that and still registers QUIC service. That can lead to accepting an h3qx-01 connection that can never make progress. Detect the failure and abort the handshake.

          _qmux_connection = std::make_unique<QMuxConnection>(this);
          this->_set_service(static_cast<QUICSupport *>(this));

Copilot

Pull request overview

Copilot reviewed 26 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/iocore/net/qmux/QMuxConnection.cc:274

When quiche_conn_recv() returns an error other than QUICHE_ERR_DONE, the code just logs and leaves the offending bytes in the read buffer. Because QMux runs over a byte stream, those bytes will be retried on every subsequent read and the connection can get stuck (and the buffer can continue to grow) instead of failing fast on a protocol error.

    if (done < 0) {
      // The record is incomplete. Leave the bytes for the next read event.
      if (done != QUICHE_ERR_DONE) {
        Dbg(dbg_ctl_qmux, "quiche_conn_recv error: %zd", done);
      }

HTTP/3 requires UDP, which is blocked or degraded on many networks. QMux carries QUIC stream multiplexing over a TLS/TCP connection so HTTP/3 can be served where UDP is unavailable. Server side only. The transport is abstracted behind the existing QUICConnection and QUICStreamIO interfaces, so HTTP/3 session and application handling is reused unchanged. QMux is offered via ALPN "h3qx-01" on TLS ports. The two transports are now selected independently of the QUIC backend. ENABLE_QUIC carries QUIC over UDP and defaults to on whenever a backend is available; ENABLE_QMUX carries it over TLS/TCP and defaults to off. Either can be enabled without the other, and QMux requires quiche built with qmux support.

AuTests need a stable feature flag to skip QMux coverage when ATS is built without the optional transport. This exposes TS_USE_QMUX through traffic_layout alongside the existing QUIC and TLS feature flags.

QMux needs an interoperable client test to prove that HTTP/3 can run over TLS/TCP and proxy multiple transactions with request and response bodies. This adds a class-based AuTest with a qmux-go client and Proxy Verifier origin. The client sends three transactions on one session, verifies forwarded headers and bodies, and checks a 300-kilobyte response byte for byte. Compatibility shims cover qmux-go v0.2.0 wire gaps.

A partial QMux record at the end of the 32 KB input buffer prevents TLS from reading the rest of the record, stalling larger request bodies. Set the input watermark to the maximum QMux record size so the buffer can append a block and complete records that span block boundaries.

Http3App's constructor runs the generic ProxySession start-up (HQSession::start()), which claims the netvc's read/write VIOs for itself. Moving qmux_con->start() before that construction, to address an earlier review comment about the app racing the transport bridge, let that claim win instead of QMuxConnection's, silently disabling QMux's connection-level I/O and crashing on the first subsequent write. Construct the app, reclaim the VIOs for QMuxConnection right after, then start the app. This keeps the app from generating stream I/O before the transport is wired up while ensuring QMuxConnection ends up owning the VIOs it depends on.

is_established() for qmux mode is qmux_transport_params_sent && qmux_transport_params_received. _handle_write() checked it before _flush_quiche_output(), which is what can flip sent to true. On the call where establishment completes this way, a stream already queued (e.g. the HTTP/3 control stream) missed its flush window, and nothing else was guaranteed to trigger another one -- if the peer waits on that stream before sending anything further, both sides stall until idle timeout. Flush once before the streams check when not yet established, so a transition to established within this call is visible to it.

ENABLE_QUICHE is a plain ON/OFF option with no AUTO state, so building with quiche never turned QMux on by itself -- ENABLE_QMUX had its own hardcoded OFF default regardless of whether the linked quiche was built with qmux support. This was the one auto_option() in the QUIC/QMux chain that didn't actually auto-detect anything, unlike ENABLE_OPENSSL_QUIC's AUTO default. quiche.h always declares quiche_config_enable_qmux() regardless of whether the library was actually built with the qmux feature, so detecting support requires a real compile-and-link check against quiche::quiche, not a header-only one -- CheckQuicheHasQmux.cmake mirrors CheckOpenSSLHasNativeQuic.cmake's shape for this reason.

ENABLE_OPENSSL_QUIC gated a capability of the mandatory OpenSSL dependency behind its own ON/OFF/AUTO option, unlike every other OpenSSL capability check in this file (SSLLIB_IS_BORINGSSL, SSLLIB_HAS_QUIC_TLS_CBS, etc.), which are plain detected variables with no option of their own. Since OpenSSL is always linked regardless, and OpenSSL-native QUIC and quiche are mutually exclusive by TLS-library requirement (quiche needs BoringSSL or the TLS callback compat shim, neither of which implements the upstream-OpenSSL-3.5+ native QUIC API), the flag never actually selected between two live backends -- disabling it had the same effect as disabling the QUIC transport outright via ENABLE_QUIC, just through a separate, asymmetric path that left a misleading "Using OpenSSL native QUIC" status line and no warning when the backend was flagged available but nothing was configured to serve it. TS_HAS_OPENSSL_QUIC is now set directly from the same detection logic, folded into the other capability checks already living in this file. The "Using ... QUIC transport" status message moves to after auto_option(QUIC ...) decides TS_USE_QUIC, so it reflects what's actually enabled rather than what's merely detected.

Copilot

Pull request overview

Copilot reviewed 27 out of 28 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

CMakeLists.txt:427

The QMux build option is declared with auto_option() but no DEFAULT is provided, so it defaults to AUTO. That enables QMux automatically whenever quiche is present and exports qmux support, which contradicts the PR description (“ENABLE_QMUX … defaults to off”).

auto_option(
  QMUX
  DESCRIPTION
  "Carry QUIC over TLS/TCP as QMux (default AUTO: on when quiche has qmux support)"
  FEATURE_VAR

quiche_conn_recv() returning anything other than QUICHE_ERR_DONE means quiche has already classified the received bytes as an unrecoverable per-connection protocol violation and started its own internal close/drain sequence internally (every non-Done error path in recv_qmux() calls self.close() before returning) -- it is never used to mean "incomplete record, wait for more bytes" in this quiche fork (both incomplete-header and incomplete-record cases are mapped to QUICHE_ERR_DONE explicitly). _handle_read() previously only logged this case and left the connection to be caught by the next scheduled quiche_conn_on_timeout() tick, which notices via quiche_conn_is_closed(). That works, but lingers for up to the connection's drain timeout doing nothing useful, and leaves the now-unparseable bytes sitting in the read buffer for that whole window. Calling close_quic_connection() immediately reaches the same end state without the wait: quiche_conn_close() is a safe no-op here since quiche already set its own close reason internally, and the pending CLOSE frame gets flushed to the peer right away instead of on the next natural write event.

Copilot

Pull request overview

Copilot reviewed 27 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

CMakeLists.txt:427

The PR description says ENABLE_QMUX “defaults to off”, but this auto_option uses the default of AUTO (which enables QMux whenever quiche+qmux support is present). Either the build default or the PR description needs to be adjusted so they match; otherwise downstream packagers/users will get QMux unexpectedly enabled when they build with a qmux-capable quiche.

auto_option(
  QMUX
  DESCRIPTION
  "Carry QUIC over TLS/TCP as QMux (default AUTO: on when quiche has qmux support)"
  FEATURE_VAR

include/iocore/net/quic/QUICStream.h:64

Making QUICStream’s destructor virtual adds a vptr to every QUICStream instance, increasing per-stream memory footprint and potentially impacting HTTP/3/QUIC performance. This class is always allocated/deleted as QUICStream (see QUICStreamManager::create_stream/delete_stream), so a virtual destructor doesn’t appear necessary for correctness here.

  virtual ~QUICStream();

maskit · 2026-08-03T08:41:46Z

@bneradt Two CMake changes:

ENABLE_QMUX now defaults to AUTO instead of OFF — it was the only one of the QUIC-related flags that didn't auto-detect, even though ENABLE_QUIC already does the equivalent (auto-on once a usable backend exists). Made it consistent.

ENABLE_OPENSSL_QUIC is gone. It turned out to be pure redundancy: OFF on that flag and OFF on ENABLE_QUIC produced the same binary either way, and it could never actually pick between two backends since OpenSSL-native QUIC and quiche require different, mutually exclusive TLS libraries — there was no build where both were even on the table to choose from.

Current user-facing flags, post-cleanup:

Flag	Controls	Default	Needs
`ENABLE_QUICHE`	pulls in the quiche library	`OFF`	—
`ENABLE_QUIC`	serve QUIC over UDP	`AUTO`	any usable backend (OpenSSL-native or quiche)
`ENABLE_QMUX`	serve QUIC over TLS/TCP	`AUTO`	quiche, built with qmux support

maskit · 2026-08-03T15:36:38Z

[approve ci autest 1]

maskit · 2026-08-03T16:15:33Z

Quiche with Qmux support: https://github.com/LPardue/quiche/tree/qmux-support

maskit added this to the 11.0.0 milestone Jul 31, 2026

maskit self-assigned this Jul 31, 2026

Copilot AI review requested due to automatic review settings July 31, 2026 16:19

maskit added QUIC HTTP/3 labels Jul 31, 2026

Copilot started reviewing on behalf of maskit July 31, 2026 16:20 View session

Copilot AI reviewed Jul 31, 2026

View reviewed changes

Comment thread src/iocore/net/SSLNetVConnection.cc

Comment thread src/iocore/net/SSLNetVConnection.cc

Comment thread src/proxy/http3/Http3SessionAccept.cc

Comment thread src/iocore/net/qmux/QMuxConnection.cc

Comment thread src/iocore/net/qmux/QMuxConnection.cc

Copilot AI review requested due to automatic review settings July 31, 2026 21:08

Copilot started reviewing on behalf of maskit July 31, 2026 21:08 View session

Copilot AI reviewed Jul 31, 2026

View reviewed changes

Comment thread tests/gold_tests/qmux/go_qmux_client/main.go

Comment thread tests/gold_tests/qmux/go_qmux_client/qmux_compat.go

Comment thread tests/gold_tests/qmux/go_qmux_client/qmux_compat.go

Comment thread tests/gold_tests/qmux/go_qmux_client/qmux_compat.go

Copilot AI review requested due to automatic review settings July 31, 2026 22:43

Copilot started reviewing on behalf of maskit July 31, 2026 22:44 View session

Copilot AI reviewed Jul 31, 2026

View reviewed changes

maskit and others added 9 commits August 3, 2026 01:04

Report QMux build support

a3babbd

AuTests need a stable feature flag to skip QMux coverage when ATS is built without the optional transport. This exposes TS_USE_QMUX through traffic_layout alongside the existing QUIC and TLS feature flags.

Address copilot comments

bfd314c

Copilot AI review requested due to automatic review settings August 3, 2026 07:22

maskit force-pushed the qmux-quiche branch from af556f3 to 72486af Compare August 3, 2026 07:22

Copilot started reviewing on behalf of maskit August 3, 2026 07:23 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

Comment thread src/iocore/net/qmux/QMuxConnection.cc

Copilot AI review requested due to automatic review settings August 3, 2026 08:26

Copilot started reviewing on behalf of maskit August 3, 2026 08:31 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

maskit added this to ATS v10.2.x Aug 3, 2026

bneradt approved these changes Aug 3, 2026

View reviewed changes

maskit merged commit bac05da into apache:master Aug 3, 2026
15 checks passed

github-project-automation Bot moved this to For v10.2.0 in ATS v10.2.x Aug 3, 2026

maskit deleted the qmux-quiche branch August 3, 2026 16:22

cmcfarlen mentioned this pull request Aug 4, 2026

Backport batch for 10.2.0 #13485

Draft

Uh oh!

Conversation

maskit commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

maskit commented Aug 3, 2026

Uh oh!

maskit commented Aug 3, 2026

Uh oh!

maskit commented Aug 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

maskit commented Jul 31, 2026 •

edited

Loading