Repository navigation
LeapFlow 0.6.0 — Production Benchmark Qualification & Reliability Hardening
Features
- Benchmark qualification suite (
leap-bench): New standalone package — manifests, registry, async runner, unified metrics, JSONL evidence, and ready/conditional/blocked gates over tier0–tier4 (plus live-llm, production-sim, pre-hardware profiles), with aleap-benchCLI (list/doctor/run/resume/report/gate) and abenchmarkinstall extra. - 36 native scenarios across 11 adapters: Refusal, approval & trust, safety policy, degradation, side-effect uncertainty, control jitter, fleet isolation, PCD efficiency, YAML onboarding, and governance — all against public APIs.
- 12 external integrations with strict official contracts: ASIMOV, AttackVLA, CALVIN, EmbodyGuard, IS-Bench, Isaac Lab, KinDER, ManiSkill, RoboJailBench, Safety-Gymnasium, VLABench, and an opt-in live-LLM harness; each returns typed UNAVAILABLE until an explicit local command and data root are configured — no auto-download, no fabricated results.
- Zero-motion Tier-4 hardware qualification with durable evidence: Preflight requires
preflight_mode: zero_motion, per-device required checks, and a structured report assertingmotion_performed=false; any motion claim or missing evidence fails closed. Every run is isolated underbenchmarks/runs/<run_id>/<adapter>/, replacing temp output. - Governed, fail-closed hardware degradation: New
DegradationCoordinatoris the single execution point for declared policies (halt/hold_position/safe_return/switch_sensor/continue_blind) across stream and health-monitor paths; latched states block writes until verified recovery or manual clear, with newhome_joint_targets,sensor_fallbacks, and boundedsafe_return_timeout_s. - Task-completion evidence and terminal failure fingerprints: A per-frame artifact ledger forces final answers once requested artifacts are confirmed;
FailureFingerprintnormalizes volatile output so repeated identical failures terminate instead of being retried blindly. - CUA window-target contract: Window screenshots require a validated
pid+window_id, malformed discovery is rejected, and bounded recovery plus a dependency circuit stop repeated RPCs on a known-bad driver — with MCP calls marshaled onto the loop that owns the stdio session. - Explicit slash-command routing and
/hostactions: Free text is only parsed as a command with a leading slash;/hostdeclares actions once for usage, completion, and both runtimes, reporting driver health, failures, and recovery steps.
Enhancements
- Failure visibility end to end: TUI renders a stable state prefix (code, retry, repeat, blocked, cooldown, exit) with next-step suggestions and tail-preserved tracebacks;
shell_runreturns structured failure codes with non-sensitive audit metadata; recovery classifies producer-declared terminal codes into a new non-recoverabletool_terminalcategory before text heuristics. - Adaptive context retention: Working memory auto-sizes from the active model window (1/8, bounded 16K–128K) instead of the legacy 8K buffer; expanded turns keep 12 recent messages and compressed summaries retain tool evidence.
- CI & packaging: Ruff and the benchmark contract suite now cover
src/benchmarks/tests/benchmarks; Tier0/Tier1 smoke runs wired into CI; manifests ship as package data; 14 new benchmark test modules plus coverage for slash routing, tool hardening, hardware degradation, and CUA mapping.
Fixes
- Daemon lifecycle: Non-child zombie processes are detected, so a stopped daemon no longer reports as alive.
- Evidence across long turns: Tool results and completion state survive tool-heavy turns and context compaction.
- Command misrouting: Natural-language input beginning with a command word no longer executes a command.
Refactor
- Degradation execution unified into
DegradationCoordinator(no more forked halt/hold/safe-return logic); repetition guard switched to normalized structured fingerprints.
What's Changed
- Production Benchmark Qualification & Reliability Hardening by @wangxingjun778 in #53
Full Changelog: v0.5.0...v0.6.0