perf: cut create-export latency by ~50% — three independent fixes

[jaredLunde]

Summary

End-to-end VM create (NATS claim → Running) dropped from ~1063ms → ~535ms on identical traces (same image, same agent-hash, same host). The "GlideFS CoW" span that was the visible long pole in Beyond's trace UI went from 415ms → 171ms — almost all of which is now the unavoidable S3 PUT for save_export.

metric	before	after	delta
GlideFS CoW span	415 ms	171 ms	−244 ms (−59%)
boot_duration_ms	584 ms	265 ms	−319 ms (−55%)
NATS claim → VM Running	1063 ms	535 ms	−528 ms (−50%)
register_device_ms	252 ms	1 ms	−251 ms
START_DEV kernel ioctl	250 ms	0.5 ms	−249 ms
Sync sysfs queue tuning	99 ms	0 ms	−99 ms
manifest_fetch_ms (warm)	123 ms	0 ms	−123 ms

The three fixes

1. Snapshot manifest cache (router.rs)

snapshots/{name}/{seq:020} is keyed by a monotonic, append-only sequence — write-once by construction, byte-identical for any given (s3_prefix, name, seq) triple. A misleading comment in the fork path claimed "snapshots are mutable" and the code refused to cache them, so every fork-from-snapshot paid a fresh ~123ms S3 GET. box-manager's ensure_derived_snapshot forks every VM from the same staging snapshot, so every single VM was re-fetching the identical bytes.

Adds a bounded HashMap<(s3_prefix, manifest_name, sequence), Arc<VolumeManifest>> next to base_manifest_cache. Pre-populates from snapshot_export so the daemon that wrote a snapshot can serve forks of it for free.

2. Background the sysfs queue-tuning writes (ublk/device.rs)

wbt_lat_usec=0 and scheduler=none were written synchronously inside register_inner, costing ~50ms each — the block layer reconfigure is surprisingly heavy on this kernel. They're tuning hints; the device is fully functional without them. spawn_blocking-ing them off the response path saves ~100ms per device-create.

3. Tick the executor before io_uring_enter (ublk/worker_pool.rs) — the big one

The biggest win and the most surprising bug. One block of code moved up bought 250ms per device-create.

The kernel's ublk_ctrl_start_dev blocks on wait_for_completion_interruptible(&ub->completion) until every queue's nr_io_ready reaches queue_depth — i.e. until every io_task has submitted its initial UBLK_IO_FETCH_REQ uring_cmd.

The worker loop order was:

1. drain inbox (handle_add_queue spawns 64 io_task futures per queue)
2. submit_with_args(to_wait=1, ...) ← blocks for up to WORKER_IDLE_NSEC = 250ms
3. drain CQEs
4. executor.tick()

io_tasks submit their FETCH_REQ SQEs on first poll — but the first poll only ran after the io_uring_enter wait. So the worker slept the full 250ms timeout waiting for CQEs that physically couldn't arrive (no SQE submitted yet), while START_DEV sat blocked on the matching completion the kernel was waiting for.

Moving the executor tick to before the submit flushes the FETCH_REQ SQEs into the ring first, the submit pushes them to the kernel immediately, ublk_mark_io_ready fires, complete_all(&ub->completion) runs, and START_DEV returns essentially instantly. The 250ms kernel ioctl is now 0.5-1ms.

How we found it

Structured timing logs at target="glidefs.timing" on each step of create_export, register_inner, the tokio::join! legs in api.rs, and inside ublk-core's start_dev (prep/wait_buf_reg/start_ioctl breakdown). These were essential to localize the 250ms — initial guesses (S3 latency, partition scan, udev blkid) were disproven by an empirical warmup experiment before we found the actual cause in the worker loop ordering.

The instrumentation stays in for ongoing observability.

Incidental cleanup

ublk-core doctests referenced libublk::* (old vendored crate name) and UblkQueue<'_> (lifetime removed in a prior refactor). 10/10 doctests now pass.

Test plan

• cargo test -p glidefs --features ublk — 893 / 893 pass
• cargo test -p ublk-core — 10 / 10 pass (including doctests)
• cargo clippy -p glidefs --features ublk --all-targets — no errors
• Live deploy via systemctl reload glidefs (zero-downtime handoff) and verified create-VM E2E timing against the homelab. Numbers in the table above are from real boxman vm logs traces.
• Watch first few production VM creates for any regression in steady-state I/O (the worker loop change has a per-iteration extra executor.tick() call which is O(woken-tasks) — cheap when no work, but worth eyeballing once.)

🤖 Generated with Claude Code