F0 GPU Sampler

F0 per-run GPU sampler

Added in 853a2fb (PR #65, issue #53, RM-1230).

tools/f0_gpu_sampler.py records NVML device state for one Agoge training run (agoge_run_id) on ShipOfTheseus and then exits. It is the collection loop that the earlier F0 pieces were built for: NVML capability discovery (bkl.f0_capability.v1), clock-skew calibration and the bkl.f0_correlation.v1 join, and the bkl.f0_efficiency.v1 summaries. It is not a daemon, not a Prometheus service, and not the trainer. Agoge still owns the process model, markers, and phase.

Spec: docs/f0-gpu-sampler.md. Human-run MiniCPM5 canary: recipes/f0-gpu-sampler.md.

CLI

python3 tools/f0_gpu_sampler.py start --run-id <id>
python3 tools/f0_gpu_sampler.py stop  --run-id <id>
python3 tools/f0_gpu_sampler.py run   --run-id <id> -- <command...>

Option	Default	Meaning
--out	results/f0-sampler	Output root (results/ is gitignored)
--cadence-ms	1000	Planned sample interval
--max-duration-s	7200	Sampler exits on its own if stop never arrives
--max-samples, --gpu-index (0), --hostname
--allow-low-headroom	off	Record a headroom breach instead of refusing (floor_held: false)
--allow-other-compute-capability	off	Skip the compute 12.0 requirement
--fake		NVML fake scenario (CPU tests)

run only brackets the argv you pass: it starts sampling, waits for that process, then stops, also when the command exits non-zero. It does not parse Agoge configs. The trainer stays in run's process group; while waiting, run ignores SIGINT (so Ctrl+C reaches the trainer once) and forwards only SIGTERM. The recipe uses start / stop with an EXIT trap instead.

Flow

flowchart TD
    S["start --run-id ID"] --> C["NVML capability snapshot<br/>(tools/nvml_capability.py)"]
    C --> I{"GPU identity present and<br/>compute capability 12.0?"}
    I -->|no| R2["refusal.json, exit 2"]
    I -->|yes| H{"free VRAM at least 2048 MiB<br/>and reading ok?"}
    H -->|"no (and no --allow-low-headroom)"| R3["refusal.json, exit 3, no samples"]
    H -->|yes| W["capability.json + session.json"]
    W --> L["sample loop every cadence_ms<br/>bkl_gpu_sample rows to samples.jsonl"]
    L -->|"stop, or --max-duration-s"| E["clock-skew.json (start/end observations)<br/>overhead.json"]
    E --> J["join with Agoge agoge-markers.jsonl<br/>check_f0_correlation.py / summarize_f0_efficiency.py"]

Loading

What one run writes

Under results/f0-sampler/<agoge_run_id>/:

File	Content
capability.json	bkl_gpu_capability snapshot, written before any sample
samples.jsonl	bkl_gpu_sample rows (bkl.f0_correlation.v1)
clock-skew.json	bkl_clock_skew from tools/f0_clock.py start and end observations
overhead.json	bkl.f0_sampler_overhead.v1 sidecar (not a join record)
session.json	pid, cadence, paths
refusal.json	only when start refuses; no sample rows are invented

A second stop is safe.

Measurements

• NVML only, through tools/nvml_live.py on the host and tools/nvml_fakes.py in CPU CI. nvidia-smi is not a metric source and no CUDA context is created. LiveNvmlBackend.begin_reading() (new in #65) clears the cached memory info so each sample reads NVML again.
• Each physical field is {value, unit, status}. unavailable, unsupported, permission_denied, transient_failure, and device_lost store value: null; a real zero stays status: ok. performance_state (unit: pstate) is recorded. cuda.runtime_version stays null and cuda.tool is nvml.
• Samples copy capability_digest, host, and GPU identity from the snapshot. profile_window_ref stays null. Phase, step, and token counts are not inferred from utilization; they come from Agoge's runs/<run_name>/telemetry/agoge-markers.jsonl (agoge-forger#147).
• After the first sample, observed_interval_ns and jitter_ns record the actual gap; collect_duration_ns is how long the reading took.

#65 also changed the efficiency summarizer: permission_denied, device_lost, and transient_failure readings are tallied as invalid and never become a peak or average (tools/f0_efficiency_numbers.py, test permission_denied_counts_as_invalid).

Overhead budget

overhead.json sets overhead_ok when the median collect_duration_ns is at most 50 ms (COLLECT_BUDGET_NS) and at most 5% of the cadence. At 1000 ms both caps are 50 ms. With no samples, overhead_ratio and overhead_ok are null.

Live NVML smoke recorded in the spec (2026-10-21, WSL2 Ubuntu 24.04, RTX 5080, Windows driver 617.14, run ... -- sleep 15, GPU near idle): 16 samples, median collect 3.02 ms (max 249 ms on the first read, NVML init), overhead_ratio 0.0030, median jitter 4.7 µs, min headroom 11357 MiB. temperature_memory reads unsupported on this driver. This is not the training-load number; that still needs the MiniCPM5 canary.

Headroom and host rules

The floor is 2048 MiB free (HEADROOM_FLOOR_MIB). The sampler does not stop the trainer and is not a lock: pause the self-hosted runner before a canary, because training and ci-gpu still serialize. sm_120 / compute 12.0 only.

CPU validation

python3 tools/check_f0_gpu_sampler.py

ci-cpu.yml runs this as the F0 GPU sampler (CPU) step. It drives the NVML fakes through full and partial support, an unavailable reading, permission denied, device lost, a real zero, no device, a headroom refusal, jitter, the duration bound, and process start/stop; sample rows are checked with tools/check_f0_correlation.py. The sampler files are in CPU_FILES in tools/gpu_ci_gate.py, so a sampler-only change does not require the GPU host.

Code

File	Role
tools/f0_gpu_sampler.py	CLI (start / stop / run), sampling loop
tools/f0_sampler_support.py	Constants, refusal codes, headroom and overhead accounting, child process handling
tools/f0_gpu_sample.py	Builds one bkl_gpu_sample row
tools/f0_sampler_harness.py	Test harness used by check_f0_gpu_sampler.py
tools/check_f0_gpu_sampler.py	CPU self-test

---

Written 2026-10-22 by Wiki Keeper (Grok Bot) from code at main @ 853a2fb.