Skip to content

Releases: sawhney17/vlabench-contract-audit

v0.1.0 — full-corpus VLABench contract audit

Choose a tag to compare

@sawhney17 sawhney17 released this 10 Aug 10:22
Immutable release. Only release title and notes can be modified.

Full-corpus VLABench contract audit and explicit-open repair

This release pins and scans all 3,114,872 action rows in
lerobot/vlabench_unified@f61ad898199e746c20af3112f278cc86eb3ca3c4 against
LeRobot's default eef environment contract at
22bd7a2f489b367d8df42de803b1e8c4ca63a3f9.

Exact findings

  • 3,114,872 / 3,114,872 rows fall outside the declared
    [-1,-1,-1,-1,-1,-1,0] to [1,1,1,1,1,1,1] Box.
  • 5,545,679 / 21,804,104 scalar action coordinates fall outside it.
  • All 10,977 episodes and 295 task strings contain at least one rejected action.
  • A broad representation-level control for absolute XYZ, Euler radians, and binary gripper
    contains every action row; it is not claimed as a calibrated physical workspace.
  • The stored state gripper bit follows a legacy closed predicate, while the action bit uses
    1 = open. Same-row complement agreement is 95.2043%; previous-action complement agreement
    is 96.2899% on comparable rows.

The trajectories are not shown to be corrupt. The pinned wrapper consumes these values as
absolute robot-frame pose targets through inverse kinematics. No rollout failure, policy-score
change, image defect, or safe silent checkpoint migration is claimed.

Repair asset

The explicit-open-v2 sidecar provides 3,114,872 keyed rows without modifying source Parquets or
actions. Consumers must explicitly choose legacy_closed_v1 or explicit_open_v2; automatic
polarity guessing is forbidden. The release includes all three sidecar shards, their manifest,
the exact audit evidence, source manifest, wheel, and source distribution.

Reproducibility

  • Reviewed commit: d5de59bcfd8be90b31cc58af8480bf92c8ed50b7.
  • 81 tests pass on Python 3.10, 3.12, and 3.13.
  • The full scan reproduces byte-identically at
    3e33d42186ca0b4058aca1551753d72f04932e362455096112adb25223316272.
  • The complete sidecar manifest is
    bdddd0f505242151f158a9652ae27a9248088acedd87d951e0c49e527b4329c1.
  • Wheel and source distributions build deterministically from the reviewed commit.

Public demo: https://sawhney17.github.io/vlabench-contract-audit/demo/

Upstream LeRobot action-contract report:
huggingface/lerobot#4406

Upstream VLABench gripper-convention report:
OpenMOSS/VLABench#88

Both reports require maintainer review. No merge or upstream acceptance is claimed.