Repository navigation
Eval integrity: taxonomy, held-out scenes, provenance
Official matches now say why they ended, which simulator ran, and which layout was used — without pretending the in-repo held-out set is secret.
Have
- Failure taxonomy on
error/result:policy.*vsprotocol.*vsharness.* - Provenance stamp: product
0.5.0, Rapier0.20.0, 60 Hz, git SHA, scene id/seed/hash - Action contract: invalid joints / huge
ee_deltanever touch Rapier (5 strikes → abort) - Latency budgets: VLA 2 s (8 strikes), state 150 ms (20 strikes) →
policy.timeout - Sparse replay artifact
vsarena-replay-v1on the result (not stored in Postgres) - Disconnect mid-match does not write ELO
Scenes
- Studio stays public canonical
- Production harness defaults to held_out (8 in-repo layouts + jitter)
- Operators can inject a private set with
VSARENA_HELD_OUT_JSON
Honest limit: those eight layouts live in git. They are a different public set, not a secret exam. True privacy needs the JSON override on the host.
Docs
- docs/eval-integrity.md
- Protocol extras on
result.failure/result.provenance/result.replay
Studio branding is v0.5.0. Python SDK 0.5.0 (parse_action_contract, format_harness_error).