Skip to content

Eval integrity: taxonomy, held-out scenes, provenance

Choose a tag to compare

@arankair arankair released this 26 Aug 20:57
· 24 commits to main since this release

Official matches now say why they ended, which simulator ran, and which layout was used — without pretending the in-repo held-out set is secret.

Have

  • Failure taxonomy on error / result: policy.* vs protocol.* vs harness.*
  • Provenance stamp: product 0.5.0, Rapier 0.20.0, 60 Hz, git SHA, scene id/seed/hash
  • Action contract: invalid joints / huge ee_delta never touch Rapier (5 strikes → abort)
  • Latency budgets: VLA 2 s (8 strikes), state 150 ms (20 strikes) → policy.timeout
  • Sparse replay artifact vsarena-replay-v1 on the result (not stored in Postgres)
  • Disconnect mid-match does not write ELO

Scenes

  • Studio stays public canonical
  • Production harness defaults to held_out (8 in-repo layouts + jitter)
  • Operators can inject a private set with VSARENA_HELD_OUT_JSON

Honest limit: those eight layouts live in git. They are a different public set, not a secret exam. True privacy needs the JSON override on the host.

Docs

Studio branding is v0.5.0. Python SDK 0.5.0 (parse_action_contract, format_harness_error).