fix(orchestrator): survive SIGBUS from failing disks under mmap'd caches

[arkamar]

A single bad 4K sector on a node's local NVMe crashed the whole
orchestrator and killed every sandbox on the node: the kernel cannot
return EIO for a memory access, so a failed page-in from an mmap'd cache
file delivers SIGBUS, which Go turns into an unrecoverable fatal error
inside the Cache.ReadAt memmove.

RunFaultSafe converts such faults into a regular error carrying the
fault address (debug.SetPanicOnFault + recover); other panics still
crash. It guards the in-process mmap copies: build segment reads, NBD
dispatch read/write/write-zeroes and the dedup compare. A bad block now
fails a single read or NBD request instead of the whole orchestrator,
and the affected file is just a local cache of remote storage anyway.
The guard is cheap enough to sit on the hot read path.

Recovered faults are logged and counted in
orchestrator.block.memory_fault, which should stay at zero on a healthy
node.

[cursor]

PR Summary

Medium Risk
Changes sit on hot NBD and dedup paths site-wide; behavior is mostly fail-fast on disk errors, but unguarded mmap reads (e.g. direct Cache.ReadAt) can still crash the process if a fault happens there.

Overview
Bad sectors under mmap'd local cache files used to raise SIGBUS and kill the orchestrator. This adds RunFaultSafe, which turns those faults into a MemoryFaultError (with fault address and an orchestrator.block.memory_fault counter) instead of a fatal runtime throw, and wraps mmap-heavy work on dedup compare, build segment reads, and NBD read/write/write-zeroes so one bad block fails a single operation or NBD request. Linux integration tests simulate truncated mmap backing files to ensure faults surface as errors and the NBD dispatch loop keeps running.

^{Reviewed by Cursor Bugbot for commit 79be12f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

❌ Patch coverage is 87.01299% with 10 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
packages/orchestrator/pkg/sandbox/block/cache.go	53.33%	6 Missing and 1 partial ⚠️
packages/orchestrator/pkg/sandbox/block/fault.go	85.00%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

[claude]

Beyond the inline finding, I checked one other fault-safety edge case and ruled it out: in nbd/dispatch.go's backing Cache.WriteAtWithoutLock, the dirty-tracker update (c.tracker.SetRange(..., Dirty)) runs only after the copy() into the mmap completes, so a fault during that copy propagates as an error before the block is ever marked Dirty — it can't be silently recorded as valid after a partial/faulted write.

Extended reasoning...

Verified directly in packages/orchestrator/pkg/sandbox/block/cache.go: the flush closure in WriteAtWithoutLock calls copy((*c.mmap)[runStart:runEnd], b[...]) before c.tracker.SetRange(startIdx, endIdx, Dirty). Since RunFaultSafe wraps the whole WriteAt call in the NBD dispatch path, a SIGBUS during that copy() unwinds via panic/recover before the subsequent SetRange line ever executes, so the block is never marked Dirty when the write only partially landed. This rules out the candidate concern that a fault-recovered write could leave an already-Dirty block silently treated as valid data.

[claude]

🟣 The P2P build-serving path (peerserver/seekable.go Stream() → ChunkService.ReadAtBuildSeekable) is not covered by the new RunFaultSafe guard: it returns a zero-copy mmap slice via diff.Slice(...) and hands it straight to sender.Send(...), where gRPC's proto marshaling performs the actual page-in memcpy outside any fault protection. A bad sector under a locally-cached build served to a peer will still raise an unrecovered SIGBUS and crash the whole orchestrator — the exact failure mode this PR fixes elsewhere — but this is pre-existing behavior in code the PR does not touch (peerserver/seekable.go, server/chunks.go), so it's not a regression introduced here.

Extended reasoning...

What the bug is: This PR introduces block.RunFaultSafe (debug.SetPanicOnFault + recover) specifically to convert SIGBUS from bad sectors under mmap'd caches into a normal error instead of a process-killing fatal throw, and applies it to build.File.readSegments, NBD read/write/write-zeroes dispatch, and the dedup compare. However, the P2P build-serving path used to share a locally-built template with peer orchestrators is not wrapped, and it performs the exact same class of raw mmap access.

The code path: Server.ReadAtBuildSeekable (packages/orchestrator/pkg/server/chunks.go) calls src.Stream(ctx, offset, length, &seekableStreamSender{stream}). seekableSource.Stream (packages/orchestrator/pkg/sandbox/template/peerserver/seekable.go) does data, _ := f.diff.Slice(ctx, offset, length, nil) and then sender.Send(data[:take]). diff.Slice resolves through StorageDiff.Slice → Chunker.Slice, whose cached/fetched fast paths return c.cache.Slice(...)/c.cache.sliceDirect(...), both of which are just (*c.mmap)[off:end] — a zero-copy slice header with no byte access at all. The actual page-in happens later, inside seekableStreamSender.Send → stream.Send(&ReadAtBuildSeekableResponse{Data: data}), where gRPC's proto marshaler copies every byte of that mmap-backed slice into the wire buffer, synchronously on the RPC handler goroutine. That goroutine never calls RunFaultSafe/debug.SetPanicOnFault.

Why existing guards don't help: Wrapping Slice itself (the way one might expect, mirroring readSegmentFaultSafe) would not fix this, since Slice never touches the mapped pages — the fault surfaces later, once the zero-copy slice escapes into gRPC's marshaler. The fix has to guard the consumer, e.g. copy the slice into a freshly allocated buffer inside RunFaultSafe before calling sender.Send.

Impact: P2P build serving is reachable whenever a template has been built locally but not yet uploaded to remote storage (the common case for freshly-built or newly-scheduled templates) — server/chunks.go only falls back to remote-storage serving once uploaded. So this is a live, non-exotic path. A bad NVMe sector under the locally-cached memfile/rootfs mmap, hit while serving a peer's ReadAtBuildSeekable request, raises an unrecovered SIGBUS and crashes the whole orchestrator process, killing every sandbox on the node — precisely the blast radius this PR sets out to eliminate.

Step-by-step proof:

1. Orchestrator A builds a template; the build's memfile diff is cached locally as a memory-mapped file, not yet uploaded to GCS.
2. Orchestrator B needs the same build and calls ChunkService.ReadAtBuildSeekable against A for a range.
3. A's Server.ReadAtBuildSeekable handler resolves the local diff and calls seekableSource.Stream.
4. Stream calls f.diff.Slice(ctx, offset, length, nil), which returns (*c.mmap)[off:end] — no memory access yet, so even if the backing block is a bad sector, nothing has faulted.
5. Stream calls sender.Send(data[:take]) → stream.Send(...); gRPC's codec marshals the Data field, reading every byte of the slice — this read hits the bad sector's mapped page and the kernel delivers SIGBUS instead of EIO (mmap has no error-return path).
6. Because this goroutine never set debug.SetPanicOnFault(true), the SIGBUS becomes an unrecoverable Go fatal runtime throw, not a catchable panic — the whole orchestrator process dies, taking every sandbox on the node down with it.

Why this is pre-existing rather than a PR regression: the PR does not touch peerserver/seekable.go or server/chunks.go, and this path was equally vulnerable before the change — the PR is still a strict improvement over the prior fully-unguarded state on the paths it does touch. It's flagged here because it's a real, in-theme completeness gap directly undercutting the PR's stated goal ("survive SIGBUS from failing disks under mmap'd caches"), and the author may want to close it in a fast follow-up by copying the slice into a fresh buffer inside RunFaultSafe before Send.

[ValentaTomas]

Good find, though, cannot the same thing happen for the Cache struct that also uses mmap?

[jakubno]

lgtm!

I added few related follow up tickets EN-2000 and EN-2003

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit edd284e. Configure here.}