v0.5.0

Six bugs, every one of them a request that never came back.

HTTP/2 retired streams by counting them. A server that answers without reading
the request body ends the stream and then resets it, and that reset lands after
the next request has gone out, so it took the new stream's place; when that
stream did end, nothing was left to notice, and it stayed in flight for the
rest of a run that could no longer finish. A POST to nginx hung about one run
in three.

HTTP/2 sent a request body and a header block whole however large they were.
SETTINGS_MAX_FRAME_SIZE was not read at all, so both could pass the 16 KiB
every peer is allowed to assume: a 20 KB body failed against haproxy, httpd,
envoy, tomcat, hypercorn, h2o, nghttpx, varnish, deno, node and bun. nginx and
caddy accept an oversized frame, which is why this lasted.

An HTTP/2 body had to fit the connection window and the stream's whole or the
request was never started, and nothing started it later - anything over 65535
bytes stopped the run with no error and no end. Bodies now leave as credit
arrives.

Any body over about 64 KiB failed on an https URL, HTTP/1.1 and HTTP/2 alike.
rustls will not take unbounded plaintext before the handshake finishes, and the
first request is handed over while it is still in flight.

QUIC threw away the material for following a key update. Either end may retire
its 1-RTT keys and say so by flipping one bit; a peer that did became a peer
whose every packet was unreadable, while the connection stayed up and the
requests stayed open. The end-to-end tests hung on this about one run in five.

Only ack-eliciting packets are in flight, and this counted every one. A packet
carrying nothing but an ACK never draws one back, so it never left the figure;
answering a large response sends a great many, and once enough had piled up to
fill the congestion window nothing ack-eliciting could go out, so the peer had
no reason to acknowledge anything. HTTP/3 against nginx hung seventeen runs in
twenty.

And two things shb would not do at all. It gave up on a Retry, so a server that
validates addresses was unreachable - none of 100 requests where h2load managed
all of them. It opened every stream a run asked for the moment the socket
connected, a round trip before the peer could say how many it would take, and
httpd, h2o and nghttpx refused 600 of 800.

New: --timeout gives up on a response after a set time, counts the request as
an error and replaces its connection, so a wedged server is a result rather
than a wait. Off by default.

Faster: HTTP/2 collects a batch of completions and writes the connection once
afterwards rather than sending from each, which is 21 % at 32 streams and
cuts kernel time per request from 11.4 to 3.0 microseconds. Received stream
data is handed over with a memcpy rather than a byte at a time, worth 24 % of
HTTP/3's userspace instructions.

Against nginx: 988k requests/sec on HTTP/1.1, 1.50M on HTTP/2, 2.31M on
HTTP/3 - 12 %, 22 % and 82 % ahead of the nearest of wrk and h2load.

The container suite grew from 73 checks to 205: every endpoint is now also
sent a request body and a large header block, two of them answer with a
quarter of a megabyte, and one demands a Retry. Four of the six bugs above
were found by those.

v0.4.1

UDP GSO now batches, which it never did.

GSO puts several datagrams in one sendmsg and the kernel splits the buffer into
equal segments, only the last allowed to be shorter. shb built datagrams back
to back and declined to batch any that came out a different size. That sounded
right and meant it never batched at all: one datagram is 1188 bytes and the
next 1190, because a packet number or a stream offset crossing a varint width
moves the total by a byte or two. Every send carried one segment, and the
machinery around it - the pinned msghdr, the control message, the probe at
start-up - was being carried for nothing.

Datagrams after the first are now built to the size of the first, which the
packet builder takes as a limit rather than as something to pad up to
afterwards. Padding afterwards cannot work: once the frames have overshot, the
packet is already encrypted. Batching begins only when the first datagram came
out nearly full, because a short one means there was nothing more to send.

With a 266 KB request body that is 28 to 64 segments per send against one
before, and 20.9 MB/s leaving the machine against 0.65. Requests per second do
not move against a server that answers before reading the body; the win is in
what goes out. Small requests still send one segment, which is correct, and
measure unchanged.

Also drops the bytes crate from the dependency list - it was there for
quinn-proto's datagram type and nothing has used it since 0.4.0 replaced the
transport - and removes what the QUIC rewrite left behind: predicates nothing
asks for, connection IDs the packet decoder was copying out of every 1-RTT
packet for no reader, and a stream lookup written out five times.

v0.4.0

shb has its own QUIC. quinn-proto is gone from the dependency list; rustls
keeps the TLS handshake, the QUIC key schedule and the packet protection,
which is the part not worth anyone rewriting.

The reason was measurement, not preference. Profiling put roughly three
quarters of shb's userspace HTTP/3 work inside quinn-proto - 38% in
handle_event, 35% in poll_transmit - against 5.6% for all of shb's own HTTP/3
code put together. quinn-proto is a complete QUIC implementation, and a
benchmark client that talks to one known peer needs a fraction of it: no server
role, no path migration, no datagram extension, one congestion controller.

Against nginx, paired interleaved runs, medians of six rounds: 338k requests
per second against 304k, and 2908ns of userspace CPU per request against
4524ns. That is 36% less work for 11% more throughput, and the gap between
those two numbers is the honest part - HTTP/3 spends only half its CPU in
userspace, and the rest is the kernel's UDP path and packet crypto, which no
amount of rewriting touches.

What it does differently: packets are built straight into the datagram buffer
the kernel will send, so a datagram costs no allocation; decoded frames borrow
from the datagram rather than taking a reference count on it; and the streams a
client opens are numbered in order, so they live in a ring indexed by
arithmetic instead of a hash map. The binary is 13% smaller.

Correctness is pinned to the specification's own worked examples - packet
numbers against RFC 9000 Appendix A.2 and A.3, initial keys and header
protection against the client Initial in RFC 9001 Appendix A.2 - and then to
other people's implementations: 58 of 58 container checks from a cold start and
22 of 22 public HTTP/3 endpoints, thirteen independent QUIC implementations in
all.

Five of this stack's bugs only ever appeared against one of them, because a
server on loopback does not drop packets, does not reorder them and sends
nothing after a stream ends. One of the five was a deliberate decision with its
reasoning written down - not buffering out-of-order handshake data, since
reordering is rare on a single path - which held right up until it met a
certificate chain spanning several packets on a real network.

v0.3.0

The released binaries are now statically linked against musl, so one binary
runs whatever glibc the machine has. The previous ones were linked against the
build runner's glibc and refused to start on anything older.

That is only worth doing because shb no longer uses musl's allocator. Measured
against nginx with paired interleaved runs, musl's own costs twice the
userspace CPU per request that glibc's does on HTTP/2 and HTTP/3. How much of
that reaches throughput depends on how much of the work is in userspace, and
the three protocols answer differently: HTTP/3 loses 36% - 292k requests/sec
down to 186k - HTTP/2 loses 4%, and HTTP/1.1 loses nothing, because 98% of its
time is already in the kernel. Every build now uses mimalloc, which closes the
gap entirely and measures as no change on glibc. cargo install
--no-default-features opts out.

Fixes a portability bug that only building for one libc had hidden:
msg_controllen and cmsg_len are size_t against glibc and socklen_t against
musl, so the HTTP/3 GSO send path did not compile for musl at all.

Also since 0.2.4, the interop suites grew to cover implementations rather than
products: Hypercorn, Node and Go for HTTP/2 decoders written independently of
the C proxies, aioquic for a fourth containerised QUIC stack, and endpoints
that reach a 48 KB header block split across CONTINUATION frames, a server
that sends GOAWAY mid-run, HTTP/2 trailers from a gRPC server, and a real 100
Continue.

v0.2.4

Correctness fixes, all of them cases where shb reported something that was
not true.

HTTP/3 no longer tears down a connection on EMSGSIZE from recv. That errno
means a datagram shb sent was too big for the path and the ICMP reply landed
on whichever operation ran next; since MTU discovery probes well past the
Ethernet default, drawing one is normal. www.bing.com failed every HTTP/3
request because of it.

HTTP/3 no longer records a 1xx as the final status, and HTTP/2 no longer
rejects trailers. Both stacks had half the rule: HTTP/3 kept the first header
section, so a 103 Early Hints buried the real response, and HTTP/2 raised an
error on a section with no :status, which is what trailers look like. All
three protocols now share one definition of an interim response.

The chunked parser no longer underflows when a read stops inside the CRLF that
ends a chunk. In release that wrapped, and the parser then waited for 2^64
more bytes.

HTTP/2 no longer asks peers to index the :authority header. The dynamic
indexing added in 0.2.2 measured no faster and broke OpenLiteSpeed, which
answered half the requests.

HTTP/2 and HTTP/3 now count a stream that ends without a :status as an error
rather than a success, so a protocol failure is reported as one.

v0.2.3

A correctness release. Both bugs were found by pointing shb at public
servers rather than at a local test server, and neither could show up
against the benchmark setup.

Fixed

The Huffman codes for '3' to '9' were off by one. RFC 7541 Appendix B
gives them 011001 through 011111; the table here started at 011010
and invented a seven-bit '9'. A Huffman-coded :status therefore decoded
one digit low wherever a 3-9 appeared — 404 came out as 303 — and a
status containing a 3 failed outright, because 011001 matched nothing.

Both HPACK and QPACK carried the same table, so HTTP/2 and HTTP/3 were
both affected. It never showed against the benchmark server because
:status 200 is a static-table index, so nothing is Huffman-decoded at
all.

TLS stopped part-way through a receive. rustls refuses more ciphertext
once 16 KiB of decrypted plaintext is waiting — one maximum-sized TLS
record — and that limit is not configurable. Feeding a whole receive before
draining any of it therefore failed against any server that sends
full-sized records: cloudflare.com, github.com and fastly.com all failed
over HTTP/2 after transferring a few hundred KiB. The plaintext is now
taken out between reads.

Also

scripts/interop.sh sends one request to each of ~50 public endpoints
across Cloudflare, Google, Meta, Fastly, Akamai, LiteSpeed, nginx, Caddy,
HAProxy and ATS, over all three protocols. All 51 pass, including HTTP/3
against five different QUIC stacks (quiche, nginx, Google, mvfst, lsquic).
It is not part of cargo test: it depends on other people's servers.

The Huffman bug survived because the tests encoded with the same table they
decoded with, so they agreed with the mistake. Everything else this project
spells out from a specification has now been checked against an independent
implementation — the HPACK and QPACK static tables, HTTP/2 and HTTP/3 frame
types, flags, stream types and settings ids, the connection preface — and
the prefix-integer and QUIC varint codecs are tested against the worked
examples in RFC 7541 C.1 and RFC 9000 A.1.

Comparison

Re-measured on the current tree. Same method as before: nginx 1.31.4 on
loopback, 10 s per run, 16 threads for every tool.

Protocol	Config	shb	wrk	h2load
HTTP/1.1	1000 connections	1,043,685	915,864	828,703
HTTP/2 (h2c)	32 conns × 32 streams	931,124	—	909,564
HTTP/2 (h2c)	100 conns × 100 streams	1,252,889	—	1,199,664
HTTP/3	32 conns × 32 streams	1,938,777	—	1,430,755

v0.2.2

A large HTTP/3 speed-up, plus a smaller HTTP/2 change.

Measured against nginx 1.31.4 on loopback, 10 s per run, 16 threads for
every tool. See the README for the full method and caveats.

Protocol	Config	shb	wrk	h2load
HTTP/1.1	1000 connections	993,170	856,476	796,238
HTTP/2 (h2c)	32 conns × 32 streams	932,839	—	885,527
HTTP/2 (h2c)	100 conns × 100 streams	1,255,321	—	1,205,942
HTTP/3	32 conns × 32 streams	1,967,413	—	1,395,173

HTTP/3: the QUIC state machine turns once per batch

Every received datagram used to run the whole drive step — poll the
connection for events, read the readable streams, open new ones, then
poll_transmit and submit the packets that came out. That costs about the
same whether one datagram arrived or eight, so doing it per datagram paid
it over and over, and it split outgoing packets across many small sends
instead of letting them collect into a GSO batch.

Datagrams now only mark their connection, and the state machines turn once
at the end of the completion batch. The gain grows as the streams per
connection drop, which is where each turn had least work to amortise it:

-c 64 -p 8 568,000 -> 966,000 +69.2%
-c 32 -p 32 1,065,000 -> 1,657,000 +57.4%
-c 16 -p 128 1,450,000 -> 1,553,000 +6.7%

Latency improves with it rather than paying for it: at 32×32 the median
goes from 1.12 ms to 0.578 ms and p99 from 1.67 ms to 1.22 ms, because the
queue a request waits in is shorter.

Against h2load this takes HTTP/3 from 26 % ahead to 48 % at 16 × 128.

HTTP/2: :authority is indexed in the peer's HPACK table

Requests were 28 bytes on the wire, 16 of them spelling out :authority
every time. Sending it once as a literal with incremental indexing and then
referring to that entry takes a request to 13 bytes. This does not measure
faster on loopback, where bytes are close to free; it is worth having for
runs over a real network, where halving the request bandwidth is not.

Also

• MSRV is 1.91.

v0.2.1

A correctness release for HTTP/1.1 response framing. No behaviour or
performance change against a server that does not send interim responses.

Fixed

1xx interim responses finished the message. An interim response — 103 Early Hints, or 100 Continue in reply to Expect: 100-continue —
completed a request and freed the connection for the next one. When the
interim and the final response arrive in one segment the counts come out
right by accident. When they arrive separately, which is how an early hint
actually arrives, every request was recorded with the interim status and
the run counted twice as many responses as it sent requests: against a
server sending 103 ahead of a 200, v0.2.0 reports 20 requests all with
status 103.

Servers that send early hints in the wild include Cloudflare and Fastly, so
anyone benchmarking through one was getting wrong numbers.

101 Switching Protocols is now rejected outright: the connection stops
being HTTP/1.1 there and a load generator has nothing to switch to.

Transfer-Encoding did not override Content-Length when chunked was
not the final coding. Transfer-Encoding: chunked, gzip alongside a
Content-Length used the Content-Length; the body actually runs to the
end of the connection.

Repeated Content-Length fields took the last value. They are only
allowed to agree; disagreeing ones are rejected now rather than framing the
stream on a guess.

Known limitation

A 2xx response to a CONNECT request is a tunnel with no body. shb treats
it as close-delimited, so such a request never completes.

v0.2.0

All three protocol stacks are now written for this one job rather than taken
from a general-purpose crate, and shb leads wrk and h2load on every protocol.

Measured against nginx 1.31.4 on loopback, 10 s per run, 16 threads for every
tool. See the README for the full method and caveats.

Protocol	Config	shb	wrk	h2load
HTTP/1.1	1000 connections	993,170	856,476	796,238
HTTP/2 (h2c)	32 conns × 32 streams	932,839	—	885,527
HTTP/2 (h2c)	100 conns × 100 streams	1,255,321	—	1,205,942
HTTP/3	32 conns × 32 streams	2,054,015	—	1,466,884

What changed

Purpose-built protocol stacks. A load generator only needs to know where
one message ends and the next begins, plus the status code to tally. All three
stacks now do exactly that:

• HTTP/1.1 responses are scanned rather than parsed: the status line,
  Content-Length, Transfer-Encoding and Connection are read, every other
  header is stepped over. Nothing is allocated per response.
• HTTP/2 and HTTP/3 requests are one HPACK/QPACK block encoded once at
  start-up from static-table references, so the encoder never touches a
  dynamic table. The client advertises SETTINGS_HEADER_TABLE_SIZE: 0 and
  QPACK_MAX_TABLE_CAPACITY: 0, which stops the peer indexing too — responses
  then decode without a dynamic table, and only :status is read.

That last point is where the HTTP/3 gain comes from: a profile of a saturated
worker used to spend 47% of its time Huffman-decoding header values nothing
looked at.

Completion batching. One io_uring_enter now covers a batch of
completions instead of one each, bounded by min_wait_usec so a batch that
cannot be filled does not stall. Needs Linux 6.12; without it each wait
returns on the first completion, as before.

--disable-keepalive, which reconnects for every request (HTTP/1.1 only).

Fixes

• MTU discovery was configured with an upper bound of 65527. That is QUIC's
  limit but not the kernel's — IPv4 cannot carry more than 65507 bytes of UDP
  payload — so the probe came back EMSGSIZE, which was treated as a dead
  connection and failed every request in flight. It cost about 4% of all
  HTTP/3 requests.

Notes

• The comparison in previous READMEs was measured against a server that
  saturated before any of the clients did, which flattered every number.
  It is now nginx with headroom, and the results changed materially.
• MSRV is 1.91.

v0.1.0

Full Changelog: https://github.com/hatoo/shb/commits/v0.1.0