feat(metrics): break down pause-snapshot latency by step

[bchalios]

Follow-up to #3425. Adds per-step snapshot latency histograms so a pause can be decomposed into where the time actually goes.

Change

• orchestrator.sandbox.snapshot.process_rootfs.duration {fs_only, success} — rootfs export+diff, recorded in pauseProcessRootfs (runs for both pause kinds; pauseOpts.filesystemSnapshot is threaded in for the label).
• orchestrator.sandbox.snapshot.process_memory.duration {success} — memory export+diff, recorded in pauseProcessMemory. Memory pauses only (fs-only skips memory processing), so no fs_only label. Times the synchronous export; the async header-dedup goroutine correctly outlives the span/timer as before.

Why

process-memory/process-rootfs existed only as trace spans, so pause latency couldn't be decomposed in metrics/dashboards. With these + pause.duration (#3425), a filesystem-only pause decomposes as:

pause.duration{fs_only="true"}                 e2e
  ├── guest_sync.duration                       quiesce (sync/freeze)
  └── process_rootfs.duration{fs_only="true"}   rootfs export/diff

and memory pauses additionally get process_memory.duration.

Notes

• Pure additive instrumentation; no pause/snapshot behavior changed.
• Follow-up: a monitoring-repo panel stacking these per-step durations (held until this deploys — the series have no data before then).

Verified: go build/vet/golangci-lint clean on changed files, go test ./pkg/sandbox/ -run TestGuestPrepareFsForPause passes.

🤖 Generated with Claude Code

[cursor]

PR Summary

Low Risk
Additive OpenTelemetry instrumentation only; pause and snapshot behavior are unchanged.

Overview
Adds histogram metrics so pause snapshot latency can be broken down in dashboards alongside existing end-to-end pause timing. Memory export and diff time is recorded during full memory pauses only, labeled by success; the timer covers the synchronous export path and does not include the async memfile header dedup work. Rootfs export and diff time is recorded for every pause, labeled by filesystem-only vs full snapshot and by success, so filesystem-only pauses can be decomposed into guest quiesce plus rootfs processing rather than relying on trace spans alone.

^{Reviewed by Cursor Bugbot for commit 832401d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

❌ Patch coverage is 0% with 18 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
packages/orchestrator/pkg/sandbox/sandbox.go	0.00%	18 Missing ⚠️

📢 Thoughts on this report? Let us know!

[claude]

LGTM — additive metrics-only change, consistent with the existing pause.duration/guest_sync.duration instrumentation pattern.

Extended reasoning...

Overview

This PR adds two new OpenTelemetry histograms (snapshot.process_memory.duration, snapshot.process_rootfs.duration) recorded inside pauseProcessMemory and pauseProcessRootfs in packages/orchestrator/pkg/sandbox/sandbox.go, plus the corresponding registration/description/unit entries in packages/shared/pkg/telemetry/meters.go. It threads the existing pauseOpts.filesystemSnapshot flag one level deeper into pauseProcessRootfs to label the new rootfs histogram with fs_only.

Security risks

None. This is pure observability instrumentation on an internal orchestrator code path — no new inputs, no auth/crypto/permission logic touched, and no data is exposed beyond duration/success/fs_only labels already exposed by the sibling pause.duration metric from the prior PR (#3425).

Level of scrutiny

Low. The change is additive-only: it wraps existing logic with a time.Now()/defer Record(...) pair and adds one new plumbed-through boolean parameter. I verified the named-return error variable (e) in both pauseProcessMemory and pauseProcessRootfs is correctly set on every return path before the deferred histogram-record closure reads it (Go assigns named returns before deferred funcs run), so the success label reflects the real outcome. No control flow, error handling, or snapshot/dedup logic was changed.

Other factors

Follows the exact pattern established by the already-merged pause.duration/guest_sync.duration metrics from #3425, so there's no new design decision here. The metric registration maps (desc/units) are updated consistently with existing entries. No test changes are included, but none are needed for pure metric-emission code, and the PR notes existing tests were run for the touched package.