echo-100k: replace upload with a 100 KB TLS echo - #1382
Merged
Conversation
upload measured ingest alone, with bodies up to 20 MB, and had stopped discriminating: 3,156 rps for humming-bird against 2,936 for go-stdlib, a 7% spread across 99 entries spanning D, Rust, Go, Ruby and C++. At 8 MB average bodies it was measuring memcpy and the loopback, not servers. in-out loads both directions at once. 100 KB up over TLS, the same 100 KB back, so every request moves 200 KB through the read path, the write path and the TLS record layer in both directions. 100 KB rather than more because in+out is already 200 KB per request, which leaves the box's bandwidth ceiling far enough away that per-request framework overhead is still visible - and it is ~7 TLS records and more than one socket buffer, so partial reads, multi-record handling and partial writes all happen on every request. The endpoint is POST /echo rather than /in-out, so a later profile can drive it at a different size or framing without a second route. Content-Length, and that is the generator's constraint rather than a preference: wrk frames the body itself and always emits Content-Length, so adding Transfer-Encoding produces a request carrying both, which RFC 9112 6.1 makes an error - wrk rejects it outright. Verified rather than assumed. Chunked moves into validation, where it is mandatory. Validation is byte-exact throughout and closes a hole the old profile had. Every upload check sent a body with an accurate Content-Length and compared the returned count, so a handler that echoed that header without reading a byte passed all of them. in-out compares the bytes: 1B, 1KB and 100KB random bodies, a chunked 100KB body that cannot be answered from a header, and an empty body. The generator rotates eight distinct 100 KB bodies rather than repeating one, so a canned response of the right size is wrong seven times in eight. Verified end to end against an echo server: 24,965 rps and all eight bodies observed in even proportion. wrk reports only the bytes it read, so its Transfer/sec is the download half of the echo. benchmark.sh reconstructs the ingest half from rps x 102400; without it the profile would publish half the I/O it moves. The 198 published upload rows are removed rather than carried over - they measure a different workload - along with the log directory and the four upload fixtures, which had no other user. The 128 entries subscribed to upload are switched to in-out. Entries do not implement POST /echo yet; that follows in this branch. Reference-only, as upload was. validate_profiles passes at 26/26 and badge parity matches at 637. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
chi, echo, fiber, gin, go-fasthttp and go-stdlib. The /upload route and its byte-count handler are replaced rather than kept beside the new one - no profile drives /upload any more. Where the framework exposes the request as a stream (net/http, echo, gin) the body is streamed straight back with Content-Length taken from the request, so nothing buffers the whole 100 KB. Where it has already been read into a buffer by the server (fiber, fasthttp) the echo is that buffer. Every one handles the chunked case explicitly: with no Content-Length the response cannot be framed until the body has been read, so those paths read first and then write. validate.sh sends a chunked 100 KB body and compares byte for byte, so this is exercised rather than assumed. All six compile. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
Rust: actix, axum, mq-bridge, ntex, rocket, salvo, trillium, trillium-tuned. JS: express, fastify, fulmine, fulmine-tuned, hyper-express, koa, node, node-h3, ultimate-express. TS: bananabread, bun, deno, elysia, hono-bun, hono-node, nestjs. Where the framework has already collected the body (axum's Bytes, ntex's Bytes, fasthttp's buffer) the echo is that buffer handed straight back at no extra copy. Where the body arrives as a stream it is collected before the response is written, which is deliberate rather than lazy: the response cannot carry a Content-Length until the length is known, and a chunked request has none to forward. validate.sh sends a chunked 100 KB body, so the streaming-through shortcut would fail it. The Web-standard runtimes (bun, deno, elysia, hono) use arrayBuffer(), which reads to end regardless of framing and gives the Response its Content-Length for free. /upload and its byte-count handler are replaced rather than left beside the new route - no profile drives it any more - and the three stale comments that explained why a JSON body parser was mounted per-route rather than globally now say /echo. Verified: axum, actix, ntex and rocket type-check clean. salvo, mq-bridge and both trilliums fail on this box for reasons unrelated to the change - a dependency wanting a different rustc, and trillium-http using unstable features - so they rest on the harness build. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
aiohttp, bjoern, blackbull, bottle, django, fastapi, fastpysgi-asgi, fastpysgi-wsgi, flask, litestar, mq-bridge-py, pyronova, robyn, sanic, slimeweb, socketify, starlette, uvicorn. Every one collects the body before replying rather than piping it through. That is the deliberate choice: the response cannot carry a Content-Length until the length is known, and a chunked request has none to forward. validate.sh posts a chunked 100 KB body and compares byte for byte, so streaming-through would fail it. Two entries needed more than the mechanical change. pyronova's upload handler used stream.drain_count(), which counts in Rust with the GIL released once and never materialises the bytes - fast for a byte count and useless for an echo, so it collects the chunks instead. bjoern's text_resp() hardcodes text/plain and takes no content type, so the echo calls make_resp() directly rather than gaining a parameter no other caller wants. slimeweb was checked rather than guessed: its response object has no raw() method. The native module exports body/bytes/file/header/html/ json/plain/send_bytes/send_text/status, and content_type is a recognised keyword, so the echo uses resp.bytes(body, content_type=...). All eighteen files compile. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
The litestar handler was renamed but its route_handlers list still named upload, which would not have imported. pyronova's max_body_size comment still explained itself in terms of a 20 MiB upload template that no longer exists. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
aspnet-minimal, aspnet-minimal-ioxide, carter, effinitive, fastendpoints,
the four genhttp variants, servicestack, simplew, simplew-tuned and sisk.
Three of these needed the framework's API checked rather than guessed,
and two of the guesses would have been wrong:
- SimpleW has no raw/binary response helper by that name. Reflecting
over the package shows HttpResponse.Body(Byte[], String) and that
HttpRequest.Body is a ReadOnlySequence<byte>, so the echo is
Response.Body(Request.Body.ToArray(), ...) - no text conversion,
which would have corrupted a binary payload.
- Effinitive's HttpRequest exposes ReadBodyAsync(ct) alongside the
CountBodyBytesAsync the old handler used; the endpoint is now
NoRequestEndpointBase<byte[]>.
- sisk's ByteArrayContent has no four-argument constructor. The build
caught it; the content type is set on the header instead.
genhttp returns a Stream rather than a byte[] so GenHTTP stays on its raw
response path instead of serializing, and the new type is named Echo to
avoid colliding with the existing EchoHandler, which is the WebSocket one.
Every one of these builds.
Two C# entries are not in this commit. ioxide counts upload bytes inside
its hand-written parser as they arrive and never buffers them, so an echo
is a real change to a hot path rather than a handler swap.
web-framework-csharp depends on WebFrameworkCSharpAPI, which is not in
the local package cache, and its SetBody is only ever called with a
string - echoing binary through one would corrupt it, so it needs the
package checked first.
Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
PHP: frankenphp-trueasync, hyperf, laravel, php, slim, swoole,
symfony-spawn-franken, symfony-spawn-tas, true-async-server, workerman.
Ruby: h2o-mruby, hanami, rage, rails, roda, sinatra.
Most were a one-line swap from a length to the bytes, because these
frameworks had already read the body to measure it. Two were not:
- true-async-server used a streaming fast path that deliberately never
materialises the body, counting chunks as they arrive. An echo needs
the bytes, so the chunks are collected instead of discarded.
- the plain php entry read $_SERVER['CONTENT_LENGTH'] and never touched
the body at all - the exact shortcut the old profile could not catch.
It now reads php://input, which reads to end regardless of framing.
hanami's action moves from Upload to Echo as its own file, since actions
there are one class per file and routed by name.
No php or ruby toolchain on this box, so these rest on the harness build
and on validate.sh's byte-exact checks.
Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
Java: helidon-production, helidon-tuned, jooby, micronaut, quarkus-jvm, spring-boot, vertx. Kotlin: fishcake, http4k, ktor, ktor-ghost. Scala: http4s, zio-http. Clojure: aleph, http-kit, pedestal, reitit, ring-http-exchange, ring-jetty-adapter, ring-jetty9-adapter. Almost all of these counted the body by transferring it to a null sink - transferTo(OutputStream.nullOutputStream()) in Java and Clojure, a fold over the chunk stream in http4s and zio-http, a Publisher subscriber accumulating a length in micronaut. None of them kept the bytes, so an echo is a real rewrite of the handler rather than a return-value change. Each now collects and returns the body, which is also what makes a chunked request work: the response cannot be framed until the length is known. Renames that follow the frameworks' conventions: helidon's UploadHandler becomes EchoHandler, spring-boot's UploadController becomes EchoController, fishcake's Upload service becomes Echo, and the two mapped ring adapters gain an echo-response beside the count-stream-bytes they no longer call. vertx needed an io.vertx.core.buffer.Buffer import that the counting version did not; micronaut swaps AtomicLong for ByteArrayOutputStream and drops the now-dead import. No maven, kotlinc, scala or clojure toolchain on this box - only gradle and javac, and these are maven projects - so these rest on the harness build and on validate.sh's byte-exact checks. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
…rue-async-server Route renames the earlier batches missed: the ktor and ktor-ghost tests still POSTed /upload and asserted a byte count (they now assert the bytes come back), rails' MarkUploadAsBinary middleware keyed on PATH_INFO '/upload', ring-jetty-adapter's 405 test named the old path, and true-async-server's own validate.sh checked for a count. web-framework-python and web-framework-csharp are the mirrored executor-per-file entries: the Upload executor is replaced by an Echo one and both registrations (executors/web.json and server/config.json) are repointed. Note on web-framework-csharp: the C++ sibling's getBody() returns a std::string, which is binary-safe, so its echo is exact. The C# API is not in the local package cache and GetHttpBody() is almost certainly a UTF-16 string, which is NOT binary-safe - a random-byte body would not survive it. The entry is disabled and in the orphan production tier, so nothing is published from it, but this needs the package checked before it is enabled. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
ioxide parses HTTP by hand, and its upload path was built for the profile it is replacing: a POST /upload was detected before the body was read, then counted as it streamed and dropped, so memory stayed bounded across 20 MB bodies. Nothing kept the bytes, so an echo could not be bolted on. The streaming counter is gone - PendingUploadRemaining, the drain branch in Feed(), the Pump() guard and FinishUpload() with it. At 100 KB there is nothing to stream around, and the body has to be buffered to be echoed anyway. The chunked decoder needed the larger change. It kept only a 256-byte peek, because its one caller wanted an integer for /baseline11 and a length for the byte count. It now also de-chunks into a per-connection buffer, so /echo has the bytes; the peek stays for the integer parse. It stops being static to reach that buffer. Verified end to end against a running server: 1 B, 1 KB and 100 KB Content-Length bodies and a chunked 100 KB body all come back byte-for-byte identical to random input, an empty body answers 200, and both the plain and chunked /baseline11 paths still return the right sum - that last one being the check that the decoder change did not break the caller it was written for. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
sark's upload endpoint bound its body as #[stream_body] BodyLen, a type that carries only a length - deliberately, for a profile that wanted a byte count off a 20 MB body. The echo needs the bytes, so it binds #[raw_body] LocalFrameBytes the way baseline_post and crud_create already do, and copies into an Owned the way the crud cache-hit path does. The BodyLen import goes with it. araara and araara-standard are unsubscribed from in-out rather than converted, because they cannot be converted from this repository: both build the server from the upstream `hcs` opam package with `opam source`, so the /upload handler is upstream code with no local copy. araara-standard applies a patch to that source, but araara has no patch mechanism at all, and writing a route rename blind against a file this repo does not contain would be a patch that fails to apply. Both are experimental and disabled, so nothing is published from either and no flagship or emerging entry is affected. They can resubscribe once upstream serves /echo. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
…ngines warp (flagship, Haskell) and fletch (emerging, Dart) are the two enabled entries the earlier sweeps missed - warp because its routes are pattern matches on a path list rather than a string literal, fletch because its handler lives under bin/ rather than src/. warp's countBody streamed and summed chunk lengths, so it kept nothing; it becomes readBody, which concatenates, and a new `octets` response builder frames the result with its own Content-Length beside the existing `plain` one. fletch collects the chunks and answers with res.bytes. hical, iris and typev are unsubscribed rather than converted. hical and iris have no source in this repository at all - their Dockerfiles `git clone` the server at build time - so there is nothing here to change, exactly like araara. typev does have source, and this is a judgement call rather than an impossibility: it is a hand-written epoll server whose upload path stream-drains the body without buffering, and whose BUFCAP and OBUFCAP are both 65536 - smaller than the 100 KB this profile echoes. Making it work means growing the buffers and reworking the drain into a read-then-write across both the Content-Length and the chunked paths, in Type-C, with no compiler on this box to catch a mistake. It is an engine entry and disabled, so the cost of guessing wrong outweighs the value. All three are disabled and none is flagship or emerging, so no scored entry is affected. Subscriptions: 126 -> 123. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
Renamed from in-out to echo-100k across the profile tables, the wrk adapter and its Lua script, the readiness probe, the CATALOG and doc map, validate.sh, the docs directory, the README and 123 meta.json files. The endpoint stays /echo, so a later profile can drive it at another size. FOUR GO ENTRIES WERE BROKEN and are fixed here: chi, echo, gin and go-stdlib streamed the body back using the request's Content-Length. That is not viable in net/http - the server drains and closes the request body as soon as response headers flush while unread body remains (chunkWriter.writeHeader, maxPostHandlerReadBytes = 256 KB), so io.Copy fails mid-way and the response is short under a Content-Length already promised, which tears the connection. The old profile's 20 MB body was over that threshold, so the body was left open and the same code worked; 100 KB is not. All four now read the body before writing anything. Verified against the running go-stdlib binary: 1 B, 1 KB, 100 KB and chunked 100 KB all byte-exact over HTTP/1.1, and twenty keep-alive requests all return exactly 102400 bytes. validate.sh now pins --http1.1 on every echo probe, and that is load-bearing rather than tidiness. Port 8081 advertises ALPN h2, so curl was negotiating HTTP/2 - where Go's h2 server does no such drain and a broken echo passes. wrk speaks only HTTP/1.1, so the profile would have benchmarked torn responses on a green validation. Isolation, checked rather than assumed. ioxide is the only entry where the change reached shared code: its chunked decoder is also what /baseline11 parses its integer body from. Every ioxide endpoint was re-run afterwards - baseline GET, POST with Content-Length, POST chunked, pipeline, json, json+br, static, static+gzip, delay, async-db, 404, Connection: close and a pipelined batch - and all still answer correctly. Helpers left orphaned by the swap are removed rather than left dangling: nestjs's countBody and count-stream-bytes in both ring adapters. Also restores CRLF on eight files that Python's universal newlines had silently converted to LF, which had rewritten three genhttp Project.cs files and servicestack whole. Those diffs are now one line each. Minor fixes from the audit: rocket drops the now-unused sink import, salvo stops copying 100 KB per request (Bytes clone is a refcount bump) and answers 400 rather than 200-with-empty-body on a read failure, and 76 framework READMEs stop documenting a /upload endpoint that returns a byte count. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
…e TLS
Five review agents went over the branch. They found real defects, not
polish, and this commit is those fixes.
Two entries did not parse at all: hono-bun and hono-node each carried a
stray `});` left by my edit, so `bun build` failed outright.
Three corrupted the body. nestjs returned a Buffer, and Nest's express
adapter does `isObject(body) ? res.json(body) : ...` - a Buffer is an
object, so 102400 bytes went out as 365887 bytes of
{"type":"Buffer","data":[...]} under an octet-stream content type. It now
writes through @res. bottle failed the chunked probe: gunicorn de-chunks
but leaves HTTP_TRANSFER_ENCODING set with no CONTENT_LENGTH, so
request.body re-parsed chunk framing and raised 400, which a bare except
turned into a 200 with an empty body; it reads wsgi.input directly now
and no longer swallows failures. genhttp-kestrel did not build - it is
pinned to GenHTTP 10.5.1, where the Method enum does not exist - so it
uses RequestMethod.Post like its siblings in the same directory. My
earlier claim that every C# entry built was wrong: twelve of thirteen did.
Two had a wrong Content-Type that byte comparison would never catch. roda
loads plain_hash_response_headers, which makes headers a case-SENSITIVE
hash, so a capitalised key did not suppress roda's lowercase text/html
default and both went out; rage's plain Hash had the same problem plus
`render plain:` overwriting it. Both now use the file's own lowercase
idiom.
Four had /echo on the wrong listener. fulmine, fulmine-tuned and elysia
registered it only on the plaintext app while :8081 got json and static;
swoole and workerman have a separate callback for :8081 entirely. The
profile drives TLS, so every one of those was a 404. The handlers are now
shared or duplicated onto the TLS side.
Scope. Seventeen entries have no way to answer on TLS :8081 - robyn has
no TLS support at all, the WebFramework family opens a single listener so
it cannot serve 8080 and 8081 together, and rage, sanic, veb and the rest
simply have no second listener. They are unsubscribed rather than given
one: adding a TLS listener to each is a different change, and several
cannot have one. 123 -> 106 subscribed.
slimeweb is implemented but flagged: its response object exposes only
plain/html/json plus set_header, with no binary responder and no content
type parameter. The header is set explicitly and the body handed to
plain(). If that build coerces to str it will not round-trip binary, and
that is the entry to watch.
Also: rocket drops an unused import, salvo stops copying 100 KB per
request and answers 400 rather than 200-with-empty-body on a read
failure, and the last /upload references in comments and READMEs are
gone from every subscribed entry.
validate_profiles passes and badge parity matches at 637.
Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
Fallout from #1375. When the `json` profile was removed its body check was kept and re-gated to `json-comp || json-h2c`, but that block probes plaintext :8080 - and an h2c-only entry has no HTTP/1.1 listener there. actix-h2c, quarkus-jvm-h2c, vanilla-h2c, wtx-http2, zix-http2 and nginx were all being failed for not answering on a port they never open. Each json profile now validates on its own port and nowhere else: json-comp on :8080, json-tls on :8081, json-h2c on :8082. json-h2c already had a complete body check of its own, so nothing is lost by dropping it from the plaintext gate - it was only ever being checked twice, once impossibly. Also closes the same anti-cheat hole in the h2c check that json-tls had: it verified total == price * quantity * m using the response's OWN price and quantity, so a fabricated item passed. It now diffs every field against data/dataset.json, matching json-comp and json-tls. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
Validation on #1382 ran 149 entries: 134 passed, 14 failed. This is those failures, and two of them were build breaks I introduced. http4k Main.kt:101 "Unresolved reference 'Status'" - the file imports Status.Companion.OK and uses the bare OK, which is what text() at line 65 already does. Docker build failed outright. jooby App.java:82 "cannot find symbol" - MediaType.octetstream does not exist in jooby. valueOf("application/octet-stream") does. Three were wrong at runtime: helidon-production, helidon-tuned 404 on :8081. The default listener's routing does not apply to a named socket, and the h1-tls socket registered only the json routes, so /echo was never bound on the port the profile drives. Registered there too. hyper-express Empty body hung the connection - uWebSockets emits neither 'data' nor 'end' for a zero-length body, so an event-driven handler never replies. Uses request.buffer(). slimeweb Unsubscribed. This is the risk flagged when it was written, and it was real: plain() rejects bytes with "argument 'resp_obj': 'bytes' object cannot be cast as 'str'", and slimeweb 0.2.6 has no other responder. The framework cannot return a binary body unchanged. The route stays as a text echo so it is ready if a release adds one. Disabled as requested: web-framework-cc, web-framework-cpp, web-framework-python, warp, fastpysgi-asgi. ktor-ghost drops fortunes, which is what it was failing on. Two failures are NOT from this branch and are left alone: humming-bird fails the json-tls TLS-quality probe by completing handshakes over tls1/tls1_1, and beskar-websocket fails its post-test health check after passing all six WebSocket assertions. Both are pre-existing and unrelated to the echo endpoint. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
Owner
Author
|
/benchmark -f ioxide -t echo-100k |
Contributor
|
👋 Benchmark request received. A collaborator will review and approve the run. |
Contributor
Benchmark ResultsFramework:
Full log |
Owner
Author
|
/benchmark -f actix -t echo-100k |
Contributor
|
👋 Benchmark request received. A collaborator will review and approve the run. |
Contributor
Benchmark ResultsFramework:
Full log |
Owner
Author
|
/benchmark -f genhttp-11 -t echo-100k |
Contributor
|
👋 Benchmark request received. A collaborator will review and approve the run. |
Contributor
Benchmark ResultsFramework:
Full log |
wrk cannot drive this profile at 32 connections: every entry benchmarked so far reported 0 rps there (genhttp-11, actix, ioxide), while 256 produced real numbers. At a 100 KB request body the per-connection write is large enough that 32 connections never fill the pipe, and the profile is measuring throughput under load rather than a connection ramp, so the low point carried no signal even when it did run. Conns 32,256 -> 256 in profiles.sh, in the CATALOG row that drives both the explorer and the scored set, and in the profile's implementation doc. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
/echo copied the 100 KB body twice: once from where it was received into the per-connection Out buffer, then again from Out into the write slab. This leaves it where it landed and has the handler write it into the slab directly behind the response header, so it is copied once - the same trick PendingStaticFd already uses for static files, with a memory span instead of a file descriptor. Header and body still leave in one flush. Measured on the real profile path (h1 over TLS, kTLS TX on, 256 connections): 575% -> ~545% CPU at equal-or-better throughput, about +8% requests per CPU percent. That matters because on the bench box ioxide runs at 6217% of its 6400% CPU allocation - it is CPU-saturated there, so CPU freed converts to throughput. The body is only left in place when nothing in the batch can disturb it first: it must be the last complete request in the buffer, so no pipelined remainder is compacted over it and no later recv slice appends onto it, and no static file may already be queued to write at the same point behind Out. Anything else falls back to the original copy. When the fast path is taken the carry buffer is retired to the pending echo and a fresh one swapped in, so a later recv cannot overwrite a body that has not been written yet; the retired buffer is recycled. Bodies under 4 KB keep copying - the bookkeeping costs more than the memcpy saves. Verified byte-exact on plaintext and TLS at 0/1/4095/4096/100K/300K bytes, chunked, pipelined, and interleaved with /pipeline, /static and the baseline route on one connection; /json, /static content negotiation, /delay and the a+b baseline are unchanged. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
128 KB x 32 instead of 16 KB x 256. That is the same 4 MB of provided-buffer ring per reactor, just carved differently: a 100 KB echo body now arrives in one recv rather than seven, which is worth about 7% requests-per-CPU on echo-100k on top of the zero-copy change. Measured neutral everywhere else it could have mattered - baseline at 512 and at 4096 connections, and json-tls at 512 - all inside run-to-run noise, and 32 buffers per reactor showed no starvation at 4096 connections. An earlier reading of mine rejected this on a 128 KB x 64 variant that did dip; x32 does not, and unlike x64 it keeps the ring size unchanged. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
The earlier sizing was picked against a load generator that was itself the bottleneck - wrk capped the box at ~50k rps while ioxide sat at 5.3 of 8 cores, so every knob looked flat and the differences between them were noise. Pinning the server to two logical CPUs makes it the constraint and the numbers separate properly. Re-measured that way (2 cores, wrk on the remaining 30, 256 connections): copy + 16 KB x 256 (the original) 13,979 rps zero-copy only 14,858 rps +6.3% recv reshape only 16,028 rps +14.7% both, at 128 KB x 32 17,117 rps +22.4% both, at 256 KB x 16 17,521 rps +25.3% So the ring reshape is the larger of the two effects, not the smaller one as the capped runs suggested. 256 KB x 16 beat 128 KB x 32 on three consecutive pairs and is still 4 MB per reactor; baseline at 4096 connections and json-tls are unchanged, and 16 buffers per reactor does not starve. Two knobs are worth recording as measured-and-rejected, because both looked harmless when the generator was capping and are not: incremental recv mode costs 15% here, and turning kTLS RX off costs 18%. Both keep their current defaults (off and on respectively). Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
Owner
Author
|
/benchmark -f ioxide -t echo-100k |
Contributor
|
👋 Benchmark request received. A collaborator will review and approve the run. |
Contributor
Benchmark ResultsFramework:
Full log |
Profile spec, the CATALOG row that drives both the explorer and the scored conn set, and the profile doc. Verified wrk holds 4096 here with no socket errors, which is worth checking given the same generator could not drive this profile at 32 connections at all (every entry reported 0 rps there). 4096 also puts the axis the profile claims to measure under real pressure. Per connection memory is flat for a streaming implementation and proportional to body size for a buffering one, so at a 100 KB body the difference between the two designs stops being a few hundred megabytes and becomes several gigabytes. Local check at 4096 connections: ioxide's RSS went 158 -> 725 MiB at a 10 KB body, and it buffers. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
Owner
Author
|
/benchmark-multiple -f ioxide,actix,ntex,genhttp-11-ioxide -t echo-100k |
Contributor
|
👋 Benchmark request received. A collaborator will review and approve the run. |
Contributor
Benchmark ResultsFrameworks: 4 | Test: ✅
|
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-100k | 4096 | 495,664 | 4633.6% | 1.1GiB | NEW | NEW |
✅ actix
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-100k | 4096 | 475,206 | 6211.1% | 339MiB | NEW | NEW |
✅ ntex
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-100k | 4096 | 520,945 | 5157.4% | 243MiB | NEW | NEW |
✅ genhttp-11-ioxide
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-100k | 4096 | 254,375 | 5959.4% | 1.3GiB | NEW | NEW |
Swaps the generator from wrk (open-loop, "how fast can this go") to zrk
(paced, "what did serving exactly this cost"). Connections 4096 -> 512, offered
rate pinned at 50,000 req/s via ZRK_RATE_ECHO_100K.
Verified end to end against ioxide before committing: 512 connections at a
50,000 target held rate_ratio 0.9889 (49,446 achieved), 395,675 requests all
2xx, zero connect/read/write/timeout errors, over TLS on 8081.
Plumbing this needed three things beyond the profile spec and the CATALOG row:
- endpoint_tool() maps echo-100k to zrk, and zrk_build_args grew an
echo-100k case (-m POST, -b @file, -k for the self-signed bench cert).
- zrk's body comes from a file, so REQUESTS_DIR is now mounted into the two
fallback ZRK_CMD paths that lacked it; without that -b @file resolves to
nothing inside the container. The fixture is generated on the host,
deterministically, so no two entries are measured against different bytes.
- the zrk image build was gated on the two latency profiles alone, so an
entry subscribing to echo-100k but not to those would have reached the run
with no generator.
One real regression, recorded in the profile doc rather than glossed: zrk takes
a single body, so the eight-body rotation that made a canned response wrong
seven times out of eight is gone. The anti-cheat now rests entirely on
validation, which posts random bodies and compares byte for byte, plus a
chunked body that cannot be sized from Content-Length at all. The benchmark run
no longer proves the bytes came back; validation does.
Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
Owner
Author
|
/benchmark-multiple -f ioxide,actix,ntex,genhttp-11-ioxide -t echo-100k |
Contributor
|
👋 Benchmark request received. A collaborator will review and approve the run. |
Contributor
Benchmark ResultsFrameworks: 4 | Test: ✅
|
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-100k | 512 | 49,396 | 638.4% | 314MiB | NEW | NEW |
✅ actix
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-100k | 512 | 49,427 | 199.1% | 92MiB | NEW | NEW |
✅ ntex
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-100k | 512 | 49,382 | 304.1% | 63MiB | NEW | NEW |
✅ genhttp-11-ioxide
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-100k | 512 | 49,355 | 304.2% | 709MiB | NEW | NEW |
Owner
Author
|
/benchmark -f ioxide -t echo-100k |
Contributor
|
👋 Benchmark request received. A collaborator will review and approve the run. |
Contributor
Benchmark ResultsFramework:
Full log |
…uffers" Back to 16 KB x 256, the value that was measured for this entry. Config.cs is now byte-identical to what it was before any of my recv changes. The earlier revert only took it as far as 128 KB x 32, which was not what was asked for. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
This reverts commit 94bd909.
Its other fourteen tests are unchanged. The /echo route stays in the entry, so resubscribing later is a one-line meta.json edit and needs no implementation work. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
Owner
Author
|
/benchmark -f fulmine -t echo-100k |
Contributor
|
👋 Benchmark request received. A collaborator will review and approve the run. |
Contributor
Benchmark ResultsFramework:
Full log |
The load body has been 10 KB since the LOH experiment; the name, the docs and the parameter tables still said 100 KB. This makes the name true rather than moving the body back. Renamed throughout: the profile key and PROFILE_ORDER, endpoint_tool, the CATALOG row and its description, ZRK_RATE_ECHO_10K, the doc directory, the wrk fixture, and the tests array in all 104 subscribing meta.json files (each re-parsed as JSON afterwards, since rebuild_site_data.py parses every one and a broken quote there breaks the whole build). Three claims in the profile doc were written for a 100 KB body and are false at 10 KB, so they are rewritten rather than renumbered: it is no longer "around seven TLS records", it does not span "more than one socket buffer", and 10 KB does emphatically leave in one write. What the profile measures at this size is per-request overhead paid twice - once in, once out - which is what the section now says. Validation gains the benchmark's own size instead of only bracketing it: byte -exact probes at 1 B, 1 KB, 10 KB and 100 KB, and the chunked probe now runs at both 10 KB and 100 KB. Keeping 100 KB matters because it is larger than anything the benchmark sends, so a handler that only works at the size it was tuned for is still caught. That check carries more weight than it used to - the paced generator sends one constant body, so validation is the only thing left that makes answering without reading impossible. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
The rate is pinned, so every entry that holds it returns the same rps and the composite cannot rank it on throughput the way it ranks the open-loop profiles. It now contributes a 0-100 score built the same way latency-1m and latency-10k build theirs: 0.60 CPU-per-request + 0.25 p99 + 0.15 p99.9, all multiplied by the fraction of the offered rate actually held, with full credit at 47,500 - 95% of the 50,000 target, since the generator never quite reaches its own number. CATALOG flags go True,True,False; infraScored stays False because scoredForType() reads it ahead of `scored` and no infrastructure entry has run this profile. Both copies of the scoring are updated, because they are two copies: LAT_FULL in site/leaderboard/index.html and FULL_RATE in scripts/latency_score.py. check_badge_parity.js compares them and fails the deploy when they drift; it passes here (637 ranks match). The part that would have silently produced a board of zeros is in benchmark.sh rather than in either scorer. Three gates named the two latency profiles explicitly, and the score's own inputs come out of them: the mktemp cgroup sampling directory that best-of-N selection reads, the fixed-rate validity check, and the block that emits cpu_usec, cpu_per_req_us, target_rate, rate_ratio and p99_9_latency onto the row. Without echo-10k in all three the rows would carry no cpu_per_req_us and every entry would have scored 0 while looking perfectly healthy. All three now include it. `latency_score.py --table --profile echo-10k` is accepted and reports no results yet, which is correct - nothing has been run and saved under the new name. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
Merged rather than rebased on purpose: rebasing 30 commits means a force-push, and a force-push at the wrong moment is what silently auto-closed PR #1363 earlier on this same work. Only two files actually conflicted. The 99 result files did not - our side deletes the dead upload-32/upload-256 rows and main's #1380 adds latency-10k-1024 rows, and git resolved those as the non-overlapping edits they are. Verified after the fact rather than assumed: 111 result files now carry latency-10k rows, and 0 still carry upload rows. README.md: main scored latency-10k in #1380 and dropped its reference-only asterisk, while our side still marked it reference-only; main's line is correct and is what survives. The Workload row takes our echo-10k line, but without the asterisk our side had on it - that mark means "excluded from the composite", and echo-10k is scored as of the previous commit. search.js is generated, so it was regenerated rather than hand-merged. Checked after merging: both CATALOG rows are scored True,True,False, LAT_FULL and FULL_RATE agree on all three fixed-rate profiles, badge parity passes at 641 ranks, and the shell scripts still parse. Claude-Session: https://claude.ai/code/session_01NsAGTadPkBtwXaYEngJomx
Owner
Author
|
/benchmark-test -t echo-10k --save |
Contributor
|
👋 Benchmark request received. A collaborator will review and approve the run. |
Contributor
Benchmark ResultsFrameworks: all subscribed to ✅
|
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,401 | 195.7% | 93MiB | NEW | NEW |
✅ aiohttp
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,381 | 770.4% | 625MiB | NEW | NEW |
✅ aleph
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,343 | 431.0% | 11.6GiB | NEW | NEW |
✅ aspnet-minimal
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,340 | 935.2% | 314MiB | NEW | NEW |
✅ axum
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,058 | 239.4% | 87MiB | NEW | NEW |
✅ blackbull
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,504 | 1022.6% | 2.1GiB | NEW | NEW |
✅ bottle
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 17,958 | 4913.3% | 2.6GiB | NEW | NEW |
✅ bun
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,357 | 401.5% | 693MiB | NEW | NEW |
✅ carter
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,315 | 878.7% | 240MiB | NEW | NEW |
✅ chi
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,107 | 1124.1% | 88MiB | NEW | NEW |
✅ deno
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 39,296 | 5064.7% | 5.0GiB | NEW | NEW |
✅ django
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 27,527 | 5152.2% | 5.3GiB | NEW | NEW |
✅ drogon
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,402 | 260.2% | 160MiB | NEW | NEW |
✅ echo
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,084 | 1147.2% | 90MiB | NEW | NEW |
✅ elysia
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,362 | 1372.0% | 1.6GiB | NEW | NEW |
✅ express
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,373 | 1147.8% | 4.7GiB | NEW | NEW |
✅ fastapi
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,117 | 1310.5% | 4.9GiB | NEW | NEW |
✅ fastendpoints
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,364 | 1053.0% | 344MiB | NEW | NEW |
✅ fastify
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,313 | 1108.5% | 3.9GiB | NEW | NEW |
✅ fastpysgi-wsgi
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,351 | 323.0% | 448MiB | NEW | NEW |
✅ fiber
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,320 | 336.8% | 83MiB | NEW | NEW |
✅ flask
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 16,688 | 4977.3% | 3.3GiB | NEW | NEW |
✅ fletch
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,289 | 4149.5% | 3.6GiB | NEW | NEW |
✅ fulmine
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,345 | 353.4% | 3.1GiB | NEW | NEW |
✅ fulmine-tuned
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,413 | 354.2% | 3.1GiB | NEW | NEW |
✅ genhttp-11
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,308 | 542.7% | 351MiB | NEW | NEW |
✅ genhttp-11-ioxide
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,348 | 319.6% | 701MiB | NEW | NEW |
✅ genhttp-11-kestrel
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,396 | 1025.9% | 392MiB | NEW | NEW |
✅ genhttp-kestrel
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,352 | 1038.5% | 322MiB | NEW | NEW |
✅ gin
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,060 | 1064.4% | 86MiB | NEW | NEW |
✅ go-fasthttp
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,346 | 291.5% | 99MiB | NEW | NEW |
✅ go-stdlib
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 48,986 | 1104.3% | 87MiB | NEW | NEW |
✅ h2o-mruby
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,204 | 260.9% | 305MiB | NEW | NEW |
✅ hanami
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,365 | 2937.4% | 5.0GiB | NEW | NEW |
✅ helidon-production
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,263 | 583.7% | 3.8GiB | NEW | NEW |
✅ helidon-tuned
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,313 | 554.8% | 2.6GiB | NEW | NEW |
✅ hono-bun
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,101 | 1093.8% | 1.7GiB | NEW | NEW |
✅ hono-node
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,251 | 2119.8% | 10.0GiB | NEW | NEW |
✅ http4k
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,325 | 546.8% | 3.4GiB | NEW | NEW |
✅ http4s
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 41,178 | 1073.5% | 6.2GiB | NEW | NEW |
✅ httpjl
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,291 | 3871.9% | 581MiB | NEW | NEW |
✅ humming-bird
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,072 | 701.2% | 78MiB | NEW | NEW |
✅ hyper-express
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,347 | 439.7% | 6.3GiB | NEW | NEW |
✅ hyperf
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 48,988 | 674.3% | 1.9GiB | NEW | NEW |
✅ jooby
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 47,453 | 512.8% | 8.8GiB | NEW | NEW |
✅ koa
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,364 | 922.2% | 2.8GiB | NEW | NEW |
✅ ktor
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,202 | 750.0% | 2.7GiB | NEW | NEW |
✅ ktor-ghost
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,249 | 694.6% | 2.9GiB | NEW | NEW |
✅ lapis
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,403 | 582.4% | 653MiB | NEW | NEW |
✅ litestar
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,407 | 1336.0% | 4.4GiB | NEW | NEW |
✅ lute
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,262 | 196.0% | 113MiB | NEW | NEW |
✅ micronaut
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,446 | 339.9% | 2.0GiB | NEW | NEW |
✅ mojolicious
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 46,385 | 4204.9% | 1016MiB | NEW | NEW |
✅ mq-bridge
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,161 | 259.4% | 88MiB | NEW | NEW |
✅ mq-bridge-py
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,369 | 883.0% | 2.9GiB | NEW | NEW |
✅ node
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,337 | 576.8% | 2.0GiB | NEW | NEW |
✅ node-h3
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,366 | 1354.7% | 4.3GiB | NEW | NEW |
✅ ntex
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,058 | 260.1% | 63MiB | NEW | NEW |
✅ oxpecker
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,351 | 1013.8% | 289MiB | NEW | NEW |
✅ phoenix-bandit
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,037 | 1936.7% | 510MiB | NEW | NEW |
✅ php
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,355 | 1474.2% | 3.8GiB | NEW | NEW |
✅ plug-cowboy
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 48,918 | 2253.0% | 592MiB | NEW | NEW |
✅ pyronova
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 28,618 | 366.6% | 902MiB | NEW | NEW |
✅ quarkus-jvm
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,395 | 764.1% | 2.3GiB | NEW | NEW |
✅ rails
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,433 | 2905.6% | 5.9GiB | NEW | NEW |
✅ reitit
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 47,243 | 531.7% | 1.3GiB | NEW | NEW |
✅ ring-http-exchange
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,304 | 636.1% | 941MiB | NEW | NEW |
✅ ring-jetty-adapter
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 47,726 | 576.5% | 1.4GiB | NEW | NEW |
✅ ring-jetty9-adapter
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 47,391 | 547.2% | 1.4GiB | NEW | NEW |
✅ roadrunner
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 12,881 | 582.8% | 307MiB | NEW | NEW |
✅ rocket
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 48,705 | 489.0% | 142MiB | NEW | NEW |
✅ roda
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,246 | 1421.8% | 7.5GiB | NEW | NEW |
✅ salvo
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 48,852 | 252.0% | 103MiB | NEW | NEW |
✅ servicestack
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 22,864 | 781.3% | 419MiB | NEW | NEW |
✅ simplew
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,381 | 480.7% | 324MiB | NEW | NEW |
✅ simplew-tuned
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,324 | 460.0% | 3.5GiB | NEW | NEW |
✅ sinatra
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,275 | 1500.1% | 7.1GiB | NEW | NEW |
✅ sisk
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,356 | 717.7% | 285MiB | NEW | NEW |
✅ spring-boot
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 48,598 | 610.5% | 1.2GiB | NEW | NEW |
✅ starlette
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,387 | 1080.0% | 3.6GiB | NEW | NEW |
✅ swerver
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,423 | 221.8% | 2.1GiB | NEW | NEW |
✅ swoole
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,356 | 282.9% | 439MiB | NEW | NEW |
✅ symfony-spawn-tas
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 48,149 | 1569.4% | 481MiB | NEW | NEW |
✅ trillium
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,058 | 244.1% | 303MiB | NEW | NEW |
✅ trillium-tuned
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,418 | 269.9% | 288MiB | NEW | NEW |
✅ true-async-server
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,363 | 324.1% | 260MiB | NEW | NEW |
✅ ultimate-express
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,358 | 554.5% | 7.4GiB | NEW | NEW |
✅ userver
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,423 | 679.2% | 250MiB | NEW | NEW |
✅ uvicorn
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,239 | 750.1% | 3.3GiB | NEW | NEW |
✅ vertx
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,247 | 305.4% | 3.3GiB | NEW | NEW |
✅ vibed
| Test | Conn | RPS | CPU | Mem | Δ RPS | Δ Mem |
|---|---|---|---|---|---|---|
| echo-10k | 512 | 49,372 | 405.0% | 225MiB | NEW | NEW |
gquintard
added a commit
to gquintard/HttpArena
that referenced
this pull request
Sep 1, 2026
upload was unscored and replaced by 8gbit upstream (MDA2AV#1373, MDA2AV#1382) before this branch's meta.json change landed, so CI correctly rejected it as an unknown profile. Removes the vmod's CountingWriter/upload_count and the /upload VCL route along with it. latency-10k already exists in the shared profile registry and drives the same GET /baseline11 baseline already validates, so subscribing costs no new code, same as latency-1m.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[skip-maintainer-ping]
Replaces the
uploadprofile withecho-100k: a 100 KB body posted over TLS and returned verbatim, so ingest and egress load at once.Why
uploadhad to goIt measured ingest alone with bodies up to 20 MB, and had stopped discriminating:
A 7% spread across 99 entries spanning D, Rust, Go, Ruby and C++ — what it looks like when a benchmark measures
memcpyand the loopback rather than the server.The profile
POST /echo· TLS:8081· 100 KB · Content-Length · conns 32/256 · wrk · reference-only100 KB in + 100 KB out is 200 KB per request, which leaves the bandwidth ceiling far enough away that per-request framework overhead is still visible, while being ~7 TLS records and more than one socket buffer — so partial reads, multi-record handling and partial writes all happen every request.
The endpoint is
/echo, not/echo-100k, so a later profile can drive it at another size without a second route.Content-Length is the generator's constraint, not a preference. wrk frames the body itself and always emits
Content-Length; addingTransfer-Encodingproduces a request carrying both, which RFC 9112 §6.1 makes an error — wrk rejects it. Verified, not assumed. Chunked moved into validation, where it is mandatory.Validation is byte-exact, and closes a hole the old profile had
Every
uploadcheck sent a body with an accurateContent-Lengthand compared the returned count — so a handler that echoed that header without reading a byte passed the entire suite.echo-100kcompares bytes: 1 B, 1 KB and 100 KB random bodies, a chunked 100 KB body that cannot be answered from a header, and an empty body.Every probe is pinned to
--http1.1, and that is load-bearing::8081advertises ALPNh2, and an echo that is broken under HTTP/1.1 can pass under h2 (see below). wrk speaks only HTTP/1.1.The generator rotates eight distinct 100 KB bodies so a canned response of the right size is wrong seven times in eight.
106 entries implemented
Every flagship and emerging entry that can serve TLS. Seventeen are unsubscribed rather than given a listener — robyn has no TLS support at all, the WebFramework family opens a single listener so it cannot serve 8080 and 8081 together, and rage, sanic, veb and the rest simply have no second listener. Five more (araara ×2, hical, iris, typev) build from sources this repo does not contain, or are hand-written servers whose buffers are smaller than the payload.
Five review agents, and they found real defects
Not polish — bugs that would have shipped:
Content-Length;net/httpdrains and closes the request body once response headers flush with unread body left (maxPostHandlerReadBytes= 256 KB), so the response went short under a promised length and tore the connection. The old 20 MB body was over that threshold, which is why the same shape worked before.});in hono-bun and hono-node.{"type":"Buffer","data":[…]}.HTTP_TRANSFER_ENCODINGset, so bottle re-parsed the framing, raised 400, and a bareexceptswallowed it.Methodenum does not exist./echoon the wrong listener (fulmine ×2, elysia, swoole, workerman) — 404 on the TLS port the profile drives.Isolation
No regressions to other endpoints. ioxide is the only entry where the change reached shared code — its chunked decoder is also what
/baseline11parses its integer body from — so every ioxide endpoint was re-run after: baseline GET, POST with Content-Length, POST chunked, pipeline, json, json+br, static, static+gzip, delay, async-db, 404,Connection: close, and a pipelined batch. All correct.I also swept all changed sources for cross-endpoint edits, which caught 8 files that had silently lost their CRLF line endings (rewriting three genhttp
Project.csfiles and both servicestack files whole). Restored.Smoke tested
15 entries started and probed live, 5/5 checks each, zero failures — all five probes over HTTP/1.1, byte-compared:
The audit agent separately live-tested express, fastify, koa, node, node-h3, bun, elysia, hono-bun, hono-node, aiohttp, blackbull, django, fastapi, flask, litestar, sanic, starlette, robyn, uvicorn and fastpysgi-asgi.
Not exercised by anyone — worth watching in this run
PHP, Ruby, JVM and the exotic tail (V, Zig, D, Lua, Luau, Perl, Julia, Swift, Elixir, Erlang, F#, C++) rest on source review plus this PR's validation. Two I would watch hardest:
plain/html/jsonplusset_header; there is no binary responder and none take a content type. If that build coerces tostr, binary will not round-trip.Verification
validate_profiles26/26,bash -nclean on every touched script,node scripts/check_badge_parity.js→ badge parity ok — 637 ranks match. Changingscripts/validate.shmeans this PR's validation covers every enabled entry, not only the ones touched.