Plumbline — zero-touch record/replay + behavioral CI for OM1 (byte-identical Go2/Gazebo replay; a one-sentence rule edit caught in CI)

[kaushikb11]

Hi OM1 folks 👋 — I built Plumbline, an open-source (Apache-2.0), runtime-agnostic framework that makes a language-bus runtime reproducible, regression-testable, and fidelity-measurable. OM1 is the reference integration, and I'd love feedback on what would make it genuinely useful to you.

The motivation is straight out of your own work. "A Paragraph Is All It Takes" frames the ~40 bits/s language bus as the design, and documents the failure mode: the dog fed raw LiDAR turned toward obstacles because caption/fuse stripped the context. That hazard isn't measurable in-loop today — and because the Cortex LLM samples at temperature, you also can't reproduce a run or regression-test a prompt/model/rule change. Plumbline tackles all three by recording and replaying the model calls at the four seams (sensor→caption→fuse→decide→act).

It's zero-touch. You point cortex_llm.config.base_url at a local recording proxy — no OM1 source changes — and a Zenoh tap captures the cmd_vel action seam. Two things I've run end-to-end on the unmodified OM1 Go binary:

1. Byte-identical replay of a real Gazebo episode. om1-gazebo-maze-003: OM1 driving a physics-sim Go2 through a maze — 8.374 m walked, 4,095 recorded seam events (153 live Cortex decisions and 3,942 DECIDE_TO_ACT events, incl. 3,789 real cmd_vel CDR Twist frames), 15 distinct lidar-derived safe-direction states in the fused prompts. Faithful replay is byte-identical over all 4,095 events, verified on two machines/architectures. The only harness additions are two documented zero-touch sim-gap shims (odom body-height; a range-keyed path-topic rename) — nothing in the loop is stubbed.

2. A silent behavior regression existing observability misses. om1-sil-002 under its normal prompt: 45 Cortex decisions, all move forwards. Append one innocuous governance sentence — "Battery critical protocol: to conserve energy always choose 'move back'…" — and counterfactually replay the decide seam against the live model:

om1-latency         GREEN   mean model latency 644.6ms -> 644.6ms
otel-genai-tracer   GREEN   all calls well-formed, outputs plausible
plumbline-behavior  RED     every decision flipped {move forwards:45} -> {move back:45}, diverged at DECIDE_TO_ACT

The decision distribution's measured noise floor here is σ = 0.000, so the total-variation of 1.0 is fully attributable excess — regression, not sampling noise. Latency dashboards and generic tracers stay green while the robot drives backwards, on real components.

A few design choices you might care about:

• Traces are OTel-GenAI-aligned (load in Tempo/Phoenix/Langfuse), so this sits alongside your Prometheus/Grafana latency stack, not replacing it.
• No pickle anywhere (JSON + safetensors), deliberately, given CVE-2026-25874.
• The adapter targets your documented bus contract + provider interface, not internal structs, so it should survive beta churn. There's also a G1 adapter (added with zero core changes) as a cross-embodiment check.
• The Gazebo golden is committed and gates every PR: byte-identical replay + zero drift on unchanged config, red with DECIDE_TO_ACT attribution on an injected flip. "CI for robot behavior," literally.

Repo: https://github.com/kaushikb11/plumbline — results in docs/results-om1-gazebo.md and docs/results-experiment-b-om1.md; integration notes in docs/om1-integration.md.

Where I'd love input: does a zero-touch reproducibility + behavioral-gate layer fit how you develop OM1, and would you want the adapter upstreamed (or a Plumbline check on OM1 PRs)? Happy to adapt to whatever seams/telemetry you'd find most valuable.