Skip to content

v0.13.0

Latest

Choose a tag to compare

@ehsanmok ehsanmok released this 23 Oct 05:56
· 16 commits to main since this release

flare v0.13.0 is a large release: 158 commits since v0.12.0. HTTP/3 now runs on N workers, and WebSocket is served on the reactor and over HTTP/3. The HTTP/2 client reuses connections, bind_tls takes SNI certificates and hot reload, and QUIC gained ECN, DATAGRAM, NEW_TOKEN, key update and path-MTU discovery. flare now needs Mojo 1.2, the thread engine is Mojo's AsyncRT pool and nothing else, and there are security fixes for QPACK, brotli and rustls. Breaking changes and behavior changes come first.

Breaking changes and behavior changes

Mojo 1.2 is required

pixi.toml and recipe.yaml now ask for mojo >=1.2.0,<2.0.0, so Mojo 1.1 no longer works. Dependencies moved with it: json v0.4.2, mojodoc v0.1.2 and mozz v0.2.2. CI uses prefix-dev/setup-pixi v0.11.2 with pixi 0.83.0. Source changes for 1.2 are std.hash for std.hashlib, RawFileDescriptor for FileDescriptor, and c_char pointers for CStringSpan.

AsyncRT is the only thread engine

-D FLARE_ASYNCRT is gone, along with the tests-asyncrt and bench-ab-threads tasks and check_asyncrt_capacity. ThreadHandle.spawn always enqueues a task on the pool the stdlib already runs on. Serving loops take a process-wide lease first: there are parallelism_level() - 1 of them, so one pool thread always stays free for short tasks. A worker that finds no lease free runs on a dedicated OS thread, as every worker did before. default_worker_count() is num_cpus() again. Per-connection WebSocket offload, watchdogs and every spawn in a forked child still use a dedicated OS thread.

Defaults that changed

  • ServerConfig.max_keepalive_requests is 1000, up from 100. At 1.5M req/s over 4096 connections, 100 meant about 15K reconnects in 5 s. 1000 matches nginx's keepalive_requests.
  • QuicServerConfig.ecn is on by default. Each connection marks its 1-RTT packets ECT(0) until its own validation fails, or until three PTOs pass with no ACK carrying counts. QuicClientConnection.start(ecn=True) is also the default on Linux.
  • HttpClient keeps HTTP/2 connections open and reuses them per origin, for both ALPN h2 and h2c prior knowledge. A pooled link that fails is retried on a fresh connection only when the request is idempotent. A proxy tunnel bypasses the pool. gRPC calls ride the same pool.
  • Http3Config.enable_connect_protocol defaults to False. It used to default to True and advertise SETTINGS_ENABLE_CONNECT_PROTOCOL while request validation always refused extended CONNECT, so every :protocol request the setting invited got a 400. The flag now drives both the advertisement and the validation (RFC 9220 section 3, RFC 8441 section 3). Set it to True to serve WebSocket over HTTP/3.
  • A server that opts into the io_uring buffer-ring loop with FLARE_BUFRING_HANDLER=1 is served by the unified loop, with a note on stderr, unless its config has every timeout at 0, no max_connections and no WebSocket upgrade. That loop arms no timers and no connection cap, so a peer could hold connections open forever.
  • QuicConnectionPool is now an alias of IdleConnPool[Http3ClientConnection]. The generic pool and the PoolableConn trait live in flare.runtime.pool.

Wire-level behavior

  • A request whose body passes Http3Config.max_body_size (10 MiB by default) is answered 413, and the client gets STOP_SENDING so it stops uploading. Before, the stream was marked as a protocol error and never answered.
  • The HTTP/3 server sends a buffered response's trailers as a final HEADERS frame. They were dropped, so gRPC's grpc-status never reached an h3 client.
  • The HTTP/2 server and client send frames no larger than the peer's SETTINGS_MAX_FRAME_SIZE, 16384 until the peer raises it. A server configured for 32 KiB frames used to send 32 KiB DATA frames to a peer that never raised the default, which is a FRAME_SIZE_ERROR.
  • Request trailers reach Request.trailers on HTTP/1.1, HTTP/2 and HTTP/3. HTTP/1.1 used to skip them, HTTP/2 merged them into the request headers (so a trailing authorization field became a header), and HTTP/3 dropped them. Fields a trailer may not carry (RFC 9110 section 6.5.1) are dropped, and a malformed HTTP/1.1 trailer line is a 400.
  • QpackDecoder.decode, decode_field_section and decode_field_section_dynamic take max_field_section_size, 64 KiB by default. See the security section.

WebSocket

  • WsProtocol is a sans-I/O server core. It takes the bytes a client sent and returns whole messages plus the bytes to send back. It enforces masking (1002), the frame and message cap before buffering (1009), CONTINUATION order (1002), UTF-8 on the reassembled text (1007), PING and PONG, a single CLOSE reply, and a bounded outbox that fails a client that stops reading with 1008. A violation queues its CLOSE and ends the session instead of raising.
  • WsUpgrade.on_reactor(handler) keeps an upgraded HTTP/1.1 connection on the worker that accepted it instead of using a thread per connection. The handler's on_message runs per reassembled message and on_close runs exactly once however the connection ends. A handler that raises ends the session with 1011. The blocking WsUpgrade(handler) path is unchanged.
  • The HTTP/2 and HTTP/3 carriers now run through WsProtocol too. They used to hand raw frames to the handler with no UTF-8 check, no fragment sequencing and no message cap, and the HTTP/2 carrier never answered a CLOSE.
  • WebSocket over HTTP/3 (RFC 9220) works on the server (HttpServer.attach_ws_h3) and the client (WsOverH3Stream.open). The client waits for the server's SETTINGS, requires extended CONNECT and a 200, and refuses a masked server frame. A tunnel keeps sending after the peer ends its side, so a CLOSE sent together with a FIN still gets its reply.
  • The h2 and h3 sidecars run on every worker. Each worker serves its own copy of the handler. attach_ws_h3 used to refuse more than one worker, and serve(handler, num_workers) silently dropped the h2 sidecar. A multi-address single-worker server now carries the h2 sidecar as well.
  • WsConnection.recv_message reassembles fragmented messages on the server, answering PING between fragments and checking UTF-8 on the whole message. This was the gap behind all 42 Autobahn failures listed for v0.12. The pinned Autobahn fuzzingclient now runs 301 cases with 294 OK, 4 NON-STRICT and 3 INFORMATIONAL, and autobahn-known-fail.txt is empty.
  • WsUpgrade.max_offloaded (4096 by default) caps the offloaded handler threads for the whole process. An upgrade past it gets a 503 before anything is detached. Before, each offloaded socket got its own detached thread with no limit.
  • Receive buffers grow geometrically and WsProtocol consumes its inbox through an offset. A 16 MiB message went from 3.9 s to 65 ms on the blocking path, and 200,000 small frames from 11.7 s to 33 ms. Before this fix, any client could make the server burn that CPU.

HTTP/3 and QUIC

New RFC coverage:

  • DATAGRAM frames (RFC 9221) and HTTP Datagrams with the Capsule Protocol (RFC 9297). flare.http3.capsule is a sans-I/O codec with a fuzz harness.
  • NEW_TOKEN (RFC 9000 section 8.1.3). The server issues an HMAC token over the client IP and issue time, valid for 24 hours, and a valid one skips the Retry. A replayed token from a spoofed address stays under the 3x amplification limit. HttpClient remembers the latest token for up to 256 origins.
  • ECN on both ends (RFC 9000 section 13.4), with validation and a congestion event on a new CE count.
  • ACK_FREQUENCY and IMMEDIATE_ACK (draft-ietf-quic-ack-frequency), with min_ack_delay advertised at 1 ms.
  • 1-RTT key update in both directions (RFC 9001 section 6). The rustls shim is now ABI 7. A forged phase bit changes nothing, because the next key is only installed after an open succeeds. The server starts an update after QuicServerConfig.key_update_interval packets, 2^23 by default.
  • Path-MTU discovery (RFC 9000 section 14, RFC 8899). Both ends start at 1200 bytes and probe up to the smaller advertised limit. They fall back to 1200 when the path drops.
  • Version Negotiation for unknown versions (RFC 9000 section 6.1). Datagrams under 1200 bytes and Version Negotiation packets themselves are never answered.
  • Pacing of 1-RTT egress (RFC 9002 section 7.7), with UDP GSO for bulk drains.
  • Graceful drain: drain(timeout_ms) sends GOAWAY on each connection's control stream, closes connections with H3_NO_ERROR once nothing is in flight, and refuses new handshakes. The h3 client records the GOAWAY, refuses new requests on that connection and stops pooling it.

HTTP/3 on N workers: QuicServerConfig.num_workers = N runs N threads, each with its own SO_REUSEPORT socket. Server connection IDs name their worker through an HMAC with a per-bind route key, so a CID does not reveal the owner. A worker that receives a packet for another worker's CID forwards it to that worker's inbox instead of sending a stateless reset, which keeps migrated and NAT-rebound connections alive. close() and drain() reach every worker.

Other additions:

  • Http3Config.stream_request_bodies delivers a request to the handler on its HEADERS, and Request.body_stream() yields the body as DATA arrives. Credit is granted only as the handler reads, so an unread body holds at most one stream window. It is off by default.
  • Http3Config.h3_datagram advertises SETTINGS_H3_DATAGRAM. QuicListener has send_http3_datagram and take_http3_datagrams.
  • QuicServerConfig.max_datagram_frame_size and QuicClientConnection.start(max_datagram_frame_size=) turn QUIC DATAGRAM on. It is off by default.
  • QuicServerConfig carries an http3: Http3Config. Before, every accepted connection was built with the defaults, so a server could not set its field-section limit, QPACK capacity, GOAWAY threshold or extended CONNECT.
  • AltSvc[Inner] middleware sets Alt-Svc: h3=":<port>"; ma=<max_age> on TCP responses so browsers can find the HTTP/3 listener. A handler that sets its own Alt-Svc keeps it.

QUIC and HTTP/3 fixes:

  • Flow control was one-sided on the server. Uploads stalled once a body passed the initial 1 MiB stream window because no MAX_STREAM_DATA was ever sent, and responses were framed in full whatever the client's limits allowed. Both are now honored.
  • Stream credit is granted from bytes delivered in order. Granting it from the highest offset received let a peer past one lost packet exceed the reassembly cap, and the server closed the connection with H3_GENERAL_PROTOCOL_ERROR.
  • The server recovers a lost handshake flight: it acknowledges Initial and Handshake packets, keeps CRYPTO chunks until acknowledged, and pads ack-eliciting Initials to 1200 bytes. It arms the PTO on every send and adds the loss-time timer.
  • Every issued connection ID is retired when a slot is reclaimed, and RETIRE_CONNECTION_ID is handled with a replacement issued. A stale CID used to route into the next connection that took the slot.
  • HANDSHAKE_DONE is sent, HTTP/3 errors go out in the application CONNECTION_CLOSE, and a draining endpoint sends no Initial or Handshake packet (RFC 9000 section 10.2.2).
  • 1-RTT packet numbers are sized to the unacked range. A fixed 2-byte encoding decodes wrongly past 2^15 packets in flight.

HTTP, TLS and runtime

  • HttpServer.bind_tls(addr, TlsServerConfig, ServerConfig) applies everything the config carries to the reactor's handshakes: mTLS, the minimum protocol version and the session-ticket policy. TlsServerConfig.min_protocol had never been applied anywhere, so an acceptor set to TLS 1.3 only still accepted TLS 1.2. The new shim call fails closed, and only TLS 1.2 and 1.3 are accepted.
  • TlsServerConfig.sni_certs takes per-host certificate pairs, exact or *.example.com, and a client with no matching SNI gets the default pair. HttpServer.reload_tls() re-reads every pair while serving. A pair is loaded and checked before anything is installed, so a bad one changes nothing. TlsSniCert is exported from flare.tls.
  • Accepted connections are balanced across workers on Linux. Each worker publishes its connection count, and a worker that has just accepted one while holding at least four, and more than one above the least-loaded peer, hands the stream to that peer through a bounded inbox and an eventfd. With four workers the busiest one used to hold 27 to 32 % of a few hundred connections. Counts now stay within one or two of each other. FLARE_BALANCE_ACCEPT=0 turns it off. serve_static joins the same group.
  • Serving workers are pinned inside the CPU set the process was started with. pin_cores used to count every online CPU, which put workers outside a taskset mask and, under a cgroup cpuset, failed silently.
  • ok_json_of[T] serializes a struct by reflection into the body buffer with no json.Value tree: 9 us against 66 us for building and dumping the equivalent Value on a 50-item payload. ok_json_value writes straight into the body buffer too. JSON request bodies parse with the simdjson backend, for example 1.3 KB in 3.1 us, down from 4.7 us.
  • HeaderMap.name_at(i) and HeaderMap.value_at(i) read fields in arrival order by reference, repeated names included (#39).
  • gzip responses are compressed with libdeflate, one compressor per level per thread. zlib stays for decompression and for permessage-deflate. An output buffer the stream does not fit is now an error on both paths, where the zlib path used to report the bytes written so far as a complete stream. Compressing JSON went from 144 us to 44 us of server CPU per request after the zlib work-buffer reuse, and to 34.9 us with libdeflate. brotli keeps its work buffers between calls.

Security

  • QPACK decoders now cap the decoded field section as well as the compressed one. An indexed line is one byte on the wire and a whole header in memory, and a dynamic-table entry can be referenced repeatedly, so a frame under the limit could expand to many times its size. This is the class disclosed by FoxIO as CVE-2025-64702 in quic-go and CVE-2026-12523 in quiche. Each line is charged name plus value plus 32 octets (RFC 9114 section 4.2.2) as it decodes, and the section is refused when it passes the cap. The request reader reports HEADERS field section above limit, and the response reader passes its remaining header budget.
  • libbrotli is required at >=1.2.1,<2 in pixi.toml and recipe.yaml. Brotli 1.0.3 through 1.2.0 has an out-of-bounds write in the decoder that a crafted stream can turn into remote code execution (CVE-2026-111497).
  • rustls is 0.23.46 and rustls-webpki is 0.103.16 in flare/tls/ffi/rustls_wrapper/Cargo.lock and benchmark/baselines/quinn/Cargo.lock (GHSA-2mjx-qc3c-rqvc, handshake messages accepted across encryption levels, fixed in 0.23.45). GitHub's Dependabot alerts for both lockfiles were still open when this was tagged, even though the lockfiles on main contain the fixed version.
  • A server at max_connections stopped accepting but left its listener registered. A client waiting in the accept queue kept the level-triggered listener readable, and the worker spun at 100 % CPU until a connection closed (#38). The listener is now taken out of the reactor at the cap and put back when a slot frees. The cap is per worker.
  • The io_uring buffer-ring loop has three fixes. A stale receive completion no longer keeps its provided buffer, which used to drain the 256-buffer ring until every receive failed with -ENOBUFS (#37). A request split across two receives no longer spins the loop. A response the socket took only partly is now finished from the send completion instead of truncated.
  • drain freed an accept-balancing group that a worker overrunning its deadline still published into. It is now leaked with that worker's stop flag and listeners.
  • An idle connection is no longer closed when its next request arrived during the same poll and a slow handler delayed the loop.
  • Reactor.register_exclusive always failed with EINVAL, because EPOLLRDHUP cannot be combined with EPOLLEXCLUSIVE. Shared-listener workers all woke for every connection. The flag is cleared now.
  • The HTTP/2 HPACK octet copy no longer goes through the UTF-8 validator, which hit a debug assert on a legal non-UTF-8 field value under ASSERT=all.

Performance

Figures come from the commit messages and docs/benchmark.md. Each step was measured once, at a different point in the cycle, so read each row as a before and after pair and do not compare numbers across rows.

Path Before After
HTTP/3 bulk download, 1 connection x 4 streams 45 MB/s 570 MB/s
HTTP/3 bulk upload, 1 connection x 4 streams 70.6 MB/s 259 MB/s
HTTP/2 over TLS, sequential HttpClient requests 683 req/s 13,509 req/s
h2c, 256 connections x 100 streams, 24 workers 650K req/s 3.81M req/s
Pipelined plaintext, 16-deep, 4096 connections, 24 workers 2.2M req/s 8.6M req/s
Static files over HTTP/2 with TLS 101K req/s 131K req/s
Static files over HTTP/3, server CPU per request 167 us 124 us
WebSocket echo, 4096 connections, 24 workers 465K msg/s 545K msg/s

On the 48-vCPU Xeon Platinum 8488C used for the README tables, HTTP/3 with 1 client and 100 streams went from 129,635 req/s (0.84x quiche) to 189,901 req/s against quiche's 152,064 (1.25x), with a lower p50 and p99.99. With num_workers of 1, 4 and 8 and 16 client threads, HTTP/3 plaintext reads 282,782, 958,690 and 1,942,413 req/s. At 4 workers over HTTP/1.1 the static path reaches 705,569 req/s against actix_web's 610,408, and the handler path reaches 607,940 req/s with p99 of 3.00 ms. On one worker nginx does 212k req/s against flare's 168k.

About 40 of the commits are copy removals and allocation cuts along the HTTP/2, QUIC, HTTP/3 and WebSocket paths. Their numbers are in the commit messages.

Build and packaging

  • New dependency: libdeflate >=1.22,<2 (MIT, on conda-forge; libdeflate0 on Debian). recipe.yaml links it and lists it under host and run. It was missing from the recipe until this release, so a package built from the recipe would not have compiled zlib_wrapper.c. The recipe also builds the wrapper with -pthread and -z nodelete, as build.sh does, because the wrapper keeps per-thread compressors with a thread-exit destructor.
  • The agent configuration moved from .cursor/ to an agent-agnostic .agent/, with symlinks for Cursor and Claude Code.
  • The .cursor/ hooks and the formal skills are shared, and .agent/AGENT.md documents the change constraints.

Formal models

The Lean models in formal/ were reduced to the code that ships and brought in line with it. Each model now has anchors that name the Mojo symbols it mirrors and a hash of their normalized source. A pre-commit gate blocks a change to a pinned symbol until its anchor is refreshed, theorem statements are locked, and differential tests run the models against the real code. New proofs cover the WebSocket protocol core against RFC 6455 (masking, fragment order, the message cap, UTF-8, the CLOSE reply and the bounded outbox) and the io_uring buffer-ring ledger. The buffer-ring receive decision moved into a sans-I/O kernel, and the proof says every completion that took a provided buffer returns it exactly once, the property that #37 broke. The Lean models are still not part of CI.

Known limits

  • QUIC pacing works from whole-millisecond RTTs, so a path under 4 ms RTT is not paced.
  • Multi-worker HTTP/3 spreads flows by 4-tuple, not by load. A packet forwarded after a migration waits for the owner's next loop turn, at most 100 ms.
  • The QPACK dynamic table is dormant. The encoder references no dynamic entry.
  • The h3 client rejects a request body larger than one packet, and get_streaming does not reach HTTP/3.
  • bind(List[SocketAddr]) is single-worker for its address cross product.
  • The io_uring buffer-ring handler path is HTTP/1.1 cleartext only and cannot stream.
  • The two rustls Dependabot alerts may still be open on GitHub (see Security).

Verification

CI passed on the last commit that touched code, 00f5636d: tests on Ubuntu and macOS, the ASan build, the lints, and conformance with h2spec and Autobahn. The tagged commit only changes docs/features.md on top of it. The compiler warnings from the Mojo 1.2 move are cleared; the one left in the log is enumeration value 'BIGINT' not handled in switch in json's simdjson wrapper. Per-feature details are in docs/features.md at this tag.

Thanks to @sano-suguru for issues #37, #38 and #39.

Full Changelog: v0.12.0...v0.13.0