Skip to content

v0.3.0

Choose a tag to compare

View all tags
@kiyeonjeon21 kiyeonjeon21 released this 16 Oct 18:53
· 15 commits to main since this release
v0.3.0
9eb94d8
This commit was signed with the committer’s verified signature.
kiyeonjeon21 Kiyeon Jeon
SSH Key Fingerprint: a/fKCcIGP8yuakra66ZYte0sK9KiyrvDd/HuNlSJBgA
Verified
Learn about vigilant mode.

pip install -U physim-gate

Behavior change: a more accurate bootstrap

Continuous metrics now use Hesterberg's expanded percentile bootstrap by default. The plain percentile interval was too narrow with small samples: with 10 trials and a true degradation exactly at the margin it said PASS or FAIL wrongly in up to 5.5% of runs per side, against a nominal 2.5%. The expanded interval brings that to at most 3.5%, and with 100 trials power is unchanged (93.7% vs 93.8%). See the measured error rates.

Intervals for continuous metrics are therefore a little wider, most noticeably below about 30 trials, and some borderline verdicts can change from PASS or FAIL to INCONCLUSIVE. Set defaults.bootstrap_method: percentile to reproduce v0.2 results. 0/1 metrics are unchanged. BCa intervals were evaluated too and did not help.

New

  • Planning for paired runs and from a pilot. physim-gate plan --rate 0.9 --margin 0.05 --discordance 0.04 plans paired 0/1 metrics (here 173 trials per run instead of 573 unpaired). physim-gate plan --from comparison.json plans every metric of an earlier comparison from its own data.
  • Baseline store. physim-gate baseline save/list/path keeps baselines as plain files under .physim-gate/baselines/; compare --baseline @main picks the stored baseline for the candidate's scenario.
  • Suites. physim-gate suite suite.yaml gates several scenarios with one exit code, per-scenario reports, one JUnit test suite per scenario and one Markdown summary.
  • Real multi-task data. examples/lerobot-metaworld/: a SmolVLA policy on four MetaWorld tasks through lerobot-eval, and examples/lerobot-suite.yaml gating both LeRobot examples.
  • Cross-checks. The statistics are tested against scipy and statsmodels.

Fixes

  • The terminal table is never truncated; when it does not fit, each metric is printed as a block.
  • Markdown tables keep intervals and verdicts on one line in narrow pull-request comments.
  • Long paths in suite errors are no longer broken across lines.

The GitHub Action (uses: kiyeonjeon21/physim-gate@v0.3.0) was exercised on a real pull request: it posts its summary and updates the same comment on later pushes.

Full changelog: CHANGELOG.md

Assets 5
Loading