feat(driver): shard the ring so cache-hit reads are not capped by one thread (backlog#1145)

[houseme]

Problem

A buffered read that hits the page cache completes inline inside io_uring_enter, so the thread driving a ring performs that read's memcpy. With one ring per driver, every cache-hit read's copy funnels through a single thread, capping the driver at one core's memory bandwidth.

Evidence (rustfs/backlog#1145, 16-core host):

• ps -L during a 1 MiB run: exactly one thread pinned at 100% CPU for the whole 8.2 s, throughput 4876 MB/s. The blocking-pool baseline hit 54701 MB/s across many threads.
• io_uring throughput was flat at ~5–8 GB/s regardless of read size — the signature of a single-thread bandwidth ceiling, not a per-op cost.
• Independent rings scaled ~linearly (1/2/4 rings → 4890 / 8969 / 15806 MB/s), pinning the driver thread as the ceiling.

Cache-miss reads are device-bound and never hit this, which is why the cold-cache sweep showed every strategy tied.

Change

UringDriver now holds N independent rings — each a shard with its own thread, pending table, backpressure semaphore, and eventfd. submit round-robins and binds the op to the chosen shard for its whole life: the ReadHandle already carried the tx/wake it must talk to, so a cancel or a deferred submission necessarily routes back to the ring whose pending table holds the op.

Every cancel-safety invariant therefore holds per shard, exactly as it did for one ring; the aggregate counters conserve because each shard's do.

• probe_and_start(entries) is unchanged — now probe_and_start_sharded(entries, 1). Additive API: no existing caller changes behavior, nobody silently grows threads.
• In-flight is capped at entries per shard, so the invariant making CQ overflow structurally unreachable is preserved per ring. The driver admits up to shards * entries concurrent reads.
• Probing happens on the first shard, so a restricted environment fails exactly as before. If a later shard fails to start, the running ones are shut down and joined before the error returns.
• Shutdown and drop ask every shard to stop before joining any, so bounded drains overlap instead of serializing shards * DRAIN_TIMEOUT.
• Still one ring set per disk → a stalled disk cannot starve another disk's rings (rustfs/backlog#1055 preserved).

Effect (warm page cache, 16-core, concurrent_pread_bench)

workload	baseline	sharded
1 MiB, conc 8	1 shard 4911 MB/s	8 shards 47361 MB/s (9.6×) — std_cached_pread is 50662
64 KiB, conc 32	std_open_pread 153678 IOPS, p999 3030 µs	8 shards 345402 IOPS, p999 897 µs
64 KiB, conc 128	std_open_pread 135155 IOPS, p999 10716 µs	8 shards 389047 IOPS, p999 4092 µs

Sharding removes the throughput deficit and keeps io_uring's tail-latency advantage, rather than trading one for the other.

Tests

Two new cases gate the sharded path: conservation + per-shard content correctness under the same mixed drop/keep stress as cancel_stress across 4 rings, and a single-shard equivalence check. A cancel routed to the wrong ring, or a pending entry lost across the split, surfaces as a conservation violation or a content mismatch.

concurrent_pread_bench takes an optional shards argument, so every number above is reproducible from this tree.

Verification

clippy --all-targets -D warnings clean; run-docker.sh both legs pass (15 tests) — leg 1 degrades gracefully (including the sharded probe), leg 2 runs real io_uring.

Follow-up (separate PR): wire ecstore to pick a shard count per disk.