Repository navigation
Throughput erodes about 23 percent from 400 to 1000 concurrent connections: what we measured, and where to look next #2194
Replies: 2 comments 1 reply
|
i know it's not the answer you're looking for but 2 suggestions:
there could easily be one or more issues here to address your core problem but it's pretty hard to drill in with so many variables |
|
Thanks, that's fair. Just to clarify the outstanding PRs involved here, because I may have made that look more complicated than it actually is: #2182, #2183, #2184 and #2185 are required just to boot Nanos on the OCI AArch64 instance. Without those four, the VM does not reach userspace. #2190 fixes the virtio-net issue that caused a data abort in vq_poll() under sustained load; without it the load test does not reliably complete. #2192 is the only networking change in the measured kernel related to the 255-entry listening queue limit: it sizes the listening socket queue according to the requested backlog. So, of the outstanding PRs present in the measured kernel, four are needed to boot on OCI AArch64, one is needed to keep virtio-net stable under load, and only #2192 changes the listening/accept queue behavior. I'll also use Docker bridge as the container reference going forward and drop the host-networking comparison, so there is one less variable in the measurements. |
|
Hi @eyberg @francescolavra the big news!!! Following up on my original question, we were able to identify the causes of both the throughput erosion above ~400 concurrent connections and the failures under overload. The important result is that neither appears to be an inherent scaling limit of Nanos. After four relatively small changes in the lwIP/Nanos networking path, Nanos now sustains its throughput up to 2,000 concurrent connections, runs without collapses, and on the same OCI A1 2 OCPU / 2 GB configuration serves more requests per second than Docker bridge. What we foundThere were two separate problems. 1. Throughput erosion: linear PCB lookup
Under this workload, a lookup visited roughly 0.9 PCBs per open connection and effectively never matched at the head of the list: clients take turns, so the next segment usually belongs to a PCB near the tail. At hundreds or thousands of concurrent connections, this made packet processing increasingly expensive. This was addressed with a hashed lookup of active PCBs: lwIP #17 — 2. Overload instability: the accept pathThe collapses, connection resets and large number of socket errors came from the accept path rather than from the PCB lookup. Three things mattered here:
No additional instrumentation was added. The kernel is the same one described in the original post, with the debug print removed. Stability improvementTests were run on OCI A1, 2 OCPU / 2 GB, using Each level used three 15-second passes on two endpoints, across four blocks on two VMs per kernel: 192 passes per kernel.
The progression was useful because it showed that the PCB lookup and overload failures were independent problems. The final 4.1 socket errors per million requests all occurred in a single pass at 2,000 connections. More importantly, with
So the original high-concurrency failure is no longer reproducible in this workload. Performance after the fixesMedian requests/s and p99 latency over 12 passes:
The difference at high concurrency is particularly interesting. At 2,000 concurrent connections:
For After the networking fixes it reaches 4,648 req/s with a 637 ms p99, essentially matching Docker's 622 ms p99 while serving substantially more requests. The scaling curve also changed. Between 400 and 1,000 connections, the fixed kernel loses only about 2–16% throughput per block, compared with 12–26% for the original kernel. What this changesThe original results made it look as if Nanos started hitting a networking scalability ceiling somewhere above 400 concurrent connections. These tests suggest something different. The degradation was caused by identifiable implementation bottlenecks in the TCP lookup and accept paths. Once those were removed, the apparent scaling ceiling moved beyond the 2,000-connection range tested here. That is probably the most interesting outcome of this investigation: fixing a few relatively small hot-path issues did not merely recover the lost performance — it allowed Nanos to remain stable under high concurrency while preserving the efficiency advantage we had already observed at lower concurrency. On this workload and VM configuration, the resulting stack now delivers both:
ReproducibilityDocker bridge and the original Nanos kernel were tested on the same OCI VMs in the same session, switching platforms by replacing the boot volume. The hashed-PCB kernel was also compared directly with the base kernel on the same VMs. The final kernel was measured in a later session. We have observed host-to-host/session variation of up to roughly 20–30%, so a final same-host A/B run would still be useful, particularly for the smaller The larger differences, especially on the lightweight endpoint, are well beyond the variation observed so far. All raw https://github.com/AngeloRubens/alifaces/tree/benchmark-rerun-2026-09-15/benchmarks See the runs under 2026-09-29 through 2026-10-04-errors. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi @eyberg @francescolavra, while preparing material for a new article to demonstrate—using real-world data—the advantages of Nanos, including in terms of performance, I came across this case.
We have been running a Jakarta EE application on nanos on Oracle Cloud aarch64 and pushing it to 1000 concurrent connections, with the same application in a container on the same shape as a reference. Getting there took a number of patches, most of which you have already merged. Two more became pull requests. A third thing we found we cannot explain, and that is what this post is about: above roughly 400 concurrent connections the throughput erodes, and we would like a hint on where to look next.
Everything below is measured. Where we are guessing, we say so.
How we got nanos running on OCI aarch64
Still open, and the four that an aarch64 OCI instance does not boot without:
Two others are open and are not in the kernel measured here, so please disregard them for this post: 2181 and 2189.
One of those open ones is why this post can exist at all. On 2 OCPU instances the kernel took a data abort in
vq_poll()after anything from a few seconds to about half an hour of load, four times, on three different images, after a fresh launch as well as after a stop and start and a reset. A kernel instrumented to print the queue state instead of faulting showed the cause: on a virtio-net transmit queue polled from one vCPU, the element at the consumer index still held the id left there one lap of the ring earlier while the device had already advanced the index past it, and a moment later the same slot held the right id. Nanos consumed the stale id, found no message in flight for it, and faulted. Leaving such an element for the next poll is #2190, and in a 60 minute A/B the fixed kernel met the condition once and kept serving. Without it none of the load described below runs to completion.What the measured kernel actually is, because this decides whether you can reproduce any of it: upstream
309fdcbplus 2182, 2183, 2184, 2185, 2190 and 2192, plus one local line that prints when the 2190 path triggers and is not proposed anywhere. 2179 is in there too but is merged, so it is not a divergence. Note that the pull requests themselves are rebased onaad473a; the binary we measured is on the older base. Without the first four the instance does not reach userspace at all, so a clean upstream tree cannot produce these numbers.The scenario, exactly
VM.Standard.A1.Flex, 2 OCPUs, 2 GB, eu-milan-1wrk -t4 -c<N> -d15s --latency --timeout 10s, three passes per level, 30 s warm-up at 100 connections/health/ready) and one that goes through CDI and serialises JSON (/api/cart)-XX:MaxRAMPercentage=80Three of the campaigns quoted here are published, with their raw files, their scripts and a generated summary:
benchmarks/2026-09-15/compares the three targets at lower connection counts,benchmarks/2026-09-21/is the unpatched sweep from 100 to 1000 connections, andbenchmarks/2026-09-22/is a kernel against kernel run of the accept drain.https://github.com/AngeloRubens/alifaces/tree/benchmark-rerun-2026-09-15
Three others are quoted here and are not published: the paired campaign that measures the listening queue change, the sweep of the patched kernel against the container, and the instrumented-kernel runs. Their numbers are inline, and we can share the console captures and the wrk output of all three.
What we hit first: the accept queue
A listening socket uses the same queue as a connected one for its pending connections,
SOCK_QUEUE_LENis 255, andlisten()is clamped to it. When that queue is fullaccept_tcp_from_lwip()returnsERR_BUFand lwIP aborts the connection, so the peer sees a reset of a connection it believed established.That reset is specific, which is what makes it usable as a signal. In the socket layer nothing else aborts a connection:
netsock_tcp_close()only ever callstcp_close(), andSO_LINGERis reported as off and is not honoured, sotcp_abortis reached from this path and from no other one we could find. A client that records a connection reset after a completed handshake is therefore recording this queue, and that is what the counting below rests on.Sizing that queue on the backlog the application asks for is what we propose in #2192. It does not remove the failure, it makes it rare, and we report below the one case where the patched kernel still refused connections. Measured on three independent pairs of instances in one session, one of each pair with the change and one without, order alternated between pairs:
/api/cart, three pairs summedThe first row needs its caveat, because what is in it is a floor and not a count.
accept_tcp_from_lwip()prints one line for every connection it refuses, and we read the console withoci compute console-history get-content --length 1000000. For the three instances without the change that came back full, at 999995, 1000001 and 1000000 bytes, each beginning in the middle of a line and none of them still holding the boot: what we have is the tail, spanning 36.9, 251.0 and 58.8 seconds of a run that lasted longer. For the three with the change the same command returned 142416, 142751 and 147734 bytes with the boot intact and nothing evicted, so there the numbers are counts. The asymmetry is itself a result: on one side a megabyte of diagnostics does not hold one run, on the other a whole run fits in a seventh of it. The 38 fall within 47 ms of each other, and the passes around them served 4059, 4106 and 4118 requests per second.On the bare servlet the requests-served difference is inside the variation we measured between machines, 17 to 19 percent, so we claim nothing there. The connections that were cut, however, are not a matter of variation: on one side they exhaust the console capture, and on the other the whole run fits in a fraction of it and holds none at all, apart from the 38 of one instance.
Where the cutting starts is the part we should have stated first, and we state it as an incidence per pass rather than as a sum of errors, because the sums invite a reading they do not support.
What makes it attributable is the design. In one session, on one instance shape, driven by one generator, the blocks were interleaved as nanos, bridge, host, host, bridge, nanos, with a fresh JVM before each block: the same application ran as a container with host networking, as a container with bridge networking, and on nanos.
wrkprints itsSocket errorsline only when one of its counters is non-zero, so a pass without that line is a pass that reported nothing, and counting passes is a statistic that no capture limit can distort.Passes that reported at least one socket error, out of the valid passes of the cell. The table is identical for
/health/readyand for/api/cart. Over the whole sweep, 48 of the 72 nanos passes reported socket errors, and 0 of the 144 container passes did, over the 19.0 million requests those 144 served.The incidence is a step, and it steps at 400: below it no pass on any of the three targets reports anything, and from it upwards every nanos pass does while no container pass ever does. That is also our answer to a fair objection, that what we are measuring is a cloud network, a NAT or our own generator. All three are present in all 216 passes, and they produce errors in none of the 144 that did not run on nanos.
How bad an affected pass is has two regimes, and they have to be separated before any of these numbers means anything. Some passes collapse: throughput falls to single digits and fourteen to sixteen thousand connections are cut in fifteen seconds. The rest hold their full rate and still lose connections, at a rate that rises with concurrency. Excluding the collapsed passes from both sides of the ratio, and counting all of wrk's socket error counters rather than one of them:
That is
/health/ready. On/api/cartthe same quantity is 0.0, 0.0, 1,359.8, 2,653.8, 2,902.3 and 4,048.2 per million. Both series are monotonic in the number of connections, and the container is 0.0 at every level of both, over the 19.0 million requests its two network modes served between them.This is also the answer to the reading that the middle of the range looks anomalous. It does, but only in the raw per level sums, and for two reasons that are ours and not the kernel's. A sum mixes the two regimes, so it mostly reports how many of its passes collapsed: the 14,447 of the 600 cell is one collapsed pass, while the 414 of the 800 cell is spread over all six of its passes, none of which collapsed, at 0, 32, 54, 67, 33 and 228. And a sum of read errors alone hides that which counter fires moves with the level: at 400 connections there are 11 read errors but 96 write errors, hence the 107 above, so quoting the 11 against the 414 compares two different things. Normalised, with all the counters and with the collapses set aside, nothing in the middle is out of line: 60.8, 109.0, 597.4, 1,613.2.
What rises with concurrency as well is how often a pass collapses at all: none at 400, one at 600, none at 800 and two at 1000 on
/health/ready, and two, one and three at 600, 800 and 1000 on/api/cart. Nine of the thirty-six passes above 400 connections collapsed. The two regimes are therefore not alternatives: the graded loss is always present and rising, and on top of it a pass sometimes falls over altogether.There is a second witness, and it is inside the guest, where no property of the network reaches: the kernel prints
accept_tcp_from_lwip queue overrunonce for every connection it refuses. On the unpatched instances of the paired campaign it printed enough of them to exhaust a megabyte of console and evict the boot, and on the patched ones the whole run fits with 0 or 38. A kernel reporting that it has run out of queue space is not an artefact of the path between two machines.We say all of this plainly because it is the part a reader might otherwise take for noise in our measurements, and because it is what 2192 mostly removes: in the paired campaign, eleven collapsed passes before and none after.
How rare it becomes is measured by a separate sweep, the one whose throughput figures the next section uses: the patched kernel alone against the container, over the same six levels, 72 passes in all. Seventy-one of them held and produced not one overrun. The last one,
/api/cartat 1000 connections, collapsed to 241 requests per second against a cell median of 3,736. Its console capture is complete, unlike the ones above: 830,133 bytes against the megabyte we asked for, beginning at the firmware and holding all three boots of that instance, with nothing evicted. The whole of it contains 12,472 overrun lines, every one of them inside a single window of 14.2 seconds at the end of the run. wrk, on the other machine, recorded 12,199 read errors for that pass.Those two numbers are counted on opposite sides of the wire by unrelated programs, and they agree to within 2.2 percent. That is the closest we can come to showing that what the client sees as a connection reset after its handshake is this queue refusing it, and it is the measurement the unpatched runs could not give us, because there the capture was exhausted before it could be read.
Two things that change does not do, so that the next section does not read as a contradiction. It makes the cutting rare rather than absent; and it does not remove the loss of throughput, which is still there with it and is the subject of the rest of this post. Nor is it a fix for the accounting we found while reading that path: a connection which dies while queued has its slot in the listen backlog released by lwIP, in
tcp_abandon()andtcp_pcb_purge(), while its entry stays in the socket queue until the application accepts it, soaccept()hands out connections that are already gone. That is #2193, with a test that fails without it. The kernel measured above carries 2192 and not 2193, and its listening queue was the deeper one, sized on the backlog the application asks for with nolisten_backlogset, so a burst of twelve thousand refusals in fourteen seconds is what a deep queue filling with entries that no longer hold a slot would look like. We say that this is consistent with the accounting, not that we measured 2193 to repair it: we did not measure it that way, and we claim no throughput benefit for it. The campaign that does measure 2193 is weaker than we would like, and its own limits are worth stating: of its six blocks, two are excluded because their consoles carry ten thousand and nine thousand lines of the 2190 diagnostic on one stuck index of the virtio used ring and both of those instances wedged. It struck one instance of each kernel, so it does not follow the change, but it leaves one pair that stayed healthy throughout, and on that pair neither kernel refused a single connection, so there was nothing there for the change to improve.Raw files for the unpatched sweep are in the repository linked above, under
benchmarks/2026-09-21/. The paired campaign and the sweep of the patched kernel are not published there: what this section quotes from them is the console captures and the wrk output, and we can share both.What remains, and is the reason for this post
With that fixed the rate still erodes with concurrency, and the container on the same shape does not. This is the sweep of the patched kernel described just above: medians of the 15 s passes of each cell, six per cell except the one that holds the collapse, which is excluded so that what is compared is the rate of a server that is working. Requests per second, with the margin over each container mode, on the bare servlet:
And on
/api/cart, where the request goes through CDI and serialises JSON:The same figures read as a loss between 400 and 1000 connections, which is the quantity this post is about:
/health/ready/api/cartSo nanos starts far ahead and does not stay there. On the bare servlet it is 50 percent ahead of host networking at 100 connections and 3 percent behind it at 1000; on
/api/cartit is 14 percent ahead at 100 and 12 percent behind at 1000. It does not break: it converges, and then it crosses. As service demand, total CPU per request (2 cores divided by throughput) goes from 101 us at 400 connections to 132 us at 1000. We are looking for about 31 us per request.What an instrumented kernel says
Counters reported on a timer, three paths bracketed with single entry and exit. Inbound packets per request measured 1.00 to 1.03, kernel counter against wrk's own tally.
The demultiplex walk in
tcp_input()overtcp_active_pcbs:What we expected to matter and does not:
wakeup_sock()low_level_output()DEFAULT_SO_RCVBUFis0x34000, same asTCP_WND; zero "incoming queue full" in six campaignsepollsrc/unix/poll.c: a notify entry per descriptor and a bitmap of the watched set, with nothing that walks that set per waittcp_fasttmr()andtcp_slowtmr()do walktcp_active_pcbs, four and two times a second, which is about 3,300 pcb visits per second against the eight million of the demultiplex walk at fifteen thousand segments a secondSo the walk grows five-fold and dominates the inbound path, and accounts for between a quarter and a half of the 31 us. We give a range because our two instrumented runs disagree: one bracket attributes 25 to 28 percent, three brackets attribute about half, and in the latter the inbound bracket contains the nested transmit time. We have not separated that cleanly yet.
The part of
tcp_input()that is not the walk also grows, 2.12 to 4.86 us per packet.tcp_input_lower()in the socket layer is short and has nothing that scales with connection count, so we suspect that residue is intcp_receive(), which walksunackedandooseqper packet. That is a guess.What we are asking
tcp_active_pcbsexpected to dominate at these connection counts, or are we holding lwIP wrong? The move-to-front heuristic says it exploits locality of segment arrivals, and a generator cycling through 1000 connections is the case with no locality. Is there something we should have configured?tcp_receive()walkingunackedandooseqa plausible place to look, or would you point elsewhere first?low_level_output()on virtio-net aarch64 in the expected range, and is it expected to be independent of the number of open connections?lwipopts.hwe should have tuned before drawing any of these conclusions?We are glad to run more instrumented campaigns and to publish the raw material that is not published yet. If any of this is a known limitation with a known answer, that is the most useful reply we could get.
All reactions