v0.4.0

Fixes the thirteen P1 findings of the v0.3.0 review.

Behavior changes

Some comparisons that passed before are now INVALID (exit code 3) or an input error (exit code 2):

• Runs that are not comparable block, whatever policy.on_invalid says: a different scenario version (under require_same_scenario, including stored @name baselines), a run with status failed, a run in which every trial failed, and shared trial ids with different recorded seeds.
• A gate in which no metric could be compared is INVALID. A gate with no required metric says so in its summary.
• The same file as baseline and candidate is an input error. Identical content from two files is only a note.
• A relative margin is undefined when the baseline values have both signs, as for a zero baseline.
• The LeRobot single-task layout (0.3.x) needs -O scenario=<name>.

See run-level checks.

Fixes

• The expanded bootstrap uses Welch degrees of freedom for two samples. With 10 candidate trials against 1,000 baseline trials, the per-side error at the margin falls from 4.3% / 4.7% to 2.2% / 2.5%. Equal samples are unchanged.
• plan --from plans an underpowered pilot with the 0/1 method and the gate's confidence.
• The GitHub Action never reports a stale verdict after an error, skips comments on pull requests from forks, and treats a failed comment as a warning.
• Long terminal summaries and notes wrap with a hanging indent.
• Releases run the whole CI workflow before publishing.

v0.3.1

Fixes three ways a comparison could produce a silently wrong verdict or exit code, found in the v0.3.0 review.

• Omitted trials now count as missing. When two runs share trial ids, a trial that only one run reports is a missing value in the other, so leaving crashed trials out can no longer turn INVALID into PASS. For physim.failure_rate such trials count as failures. Runs with disjoint trial ids are still compared as independent samples.
• Duplicate keys are rejected in gate configs, suites, run results and LeRobot output, instead of silently keeping the last value. Errors name the file.
• An adapter calling sys.exit() is an error (exit code 2). Previously sys.exit(0) ended the process with exit code 0, which CI reads as a passing gate.

Comparisons that relied on omitted trials being ignored may now be INVALID; see missing trials.

v0.3.0

pip install -U physim-gate

Behavior change: a more accurate bootstrap

Continuous metrics now use Hesterberg's expanded percentile bootstrap by default. The plain percentile interval was too narrow with small samples: with 10 trials and a true degradation exactly at the margin it said PASS or FAIL wrongly in up to 5.5% of runs per side, against a nominal 2.5%. The expanded interval brings that to at most 3.5%, and with 100 trials power is unchanged (93.7% vs 93.8%). See the measured error rates.

Intervals for continuous metrics are therefore a little wider, most noticeably below about 30 trials, and some borderline verdicts can change from PASS or FAIL to INCONCLUSIVE. Set defaults.bootstrap_method: percentile to reproduce v0.2 results. 0/1 metrics are unchanged. BCa intervals were evaluated too and did not help.

New

• Planning for paired runs and from a pilot. physim-gate plan --rate 0.9 --margin 0.05 --discordance 0.04 plans paired 0/1 metrics (here 173 trials per run instead of 573 unpaired). physim-gate plan --from comparison.json plans every metric of an earlier comparison from its own data.
• Baseline store. physim-gate baseline save/list/path keeps baselines as plain files under .physim-gate/baselines/; compare --baseline @main picks the stored baseline for the candidate's scenario.
• Suites. physim-gate suite suite.yaml gates several scenarios with one exit code, per-scenario reports, one JUnit test suite per scenario and one Markdown summary.
• Real multi-task data. examples/lerobot-metaworld/: a SmolVLA policy on four MetaWorld tasks through lerobot-eval, and examples/lerobot-suite.yaml gating both LeRobot examples.
• Cross-checks. The statistics are tested against scipy and statsmodels.

Fixes

• The terminal table is never truncated; when it does not fit, each metric is printed as a block.
• Markdown tables keep intervals and verdicts on one line in narrow pull-request comments.
• Long paths in suite errors are no longer broken across lines.

The GitHub Action (uses: kiyeonjeon21/physim-gate@v0.3.0) was exercised on a real pull request: it posts its summary and updates the same comment on later pushes.

Full changelog: CHANGELOG.md

v0.2.0

PhySim Gate is now on PyPI: pip install physim-gate.

Highlights

• LeRobot adapter. --adapter lerobot reads the eval_info.json that lerobot-eval writes (LeRobot 0.4.0+ per_task layout and the 0.3.x per_episode layout). Evaluations run with the same --seed are compared episode by episode.
• Real simulator example. examples/lerobot-pusht/ gates unedited lerobot-eval output from the PushT simulator (100 episodes per run): a rerun of the same policy passes; changing the action horizon fails on max_reward, while the success drop from 11% to 1% is honestly INCONCLUSIVE at 100 episodes.
• physim-gate plan. How many trials does a verdict need? Exact verdict probabilities for 0/1 metrics using the engine's own score interval; a normal approximation for continuous metrics. Example: showing that a 90% success rate has not dropped by more than 5 points needs about 573 episodes per run.
• Markdown summaries. --markdown FILE and --format markdown for pull-request comments and job summaries.
• GitHub Action. uses: kiyeonjeon21/physim-gate@v0.2.0 runs the gate, writes all reports and the job summary, exposes verdict and exit-code, and can post or update a pull-request comment.

Fixes

• Paths in the "wrote ..." line are no longer wrapped mid-name in narrow or redirected terminals.

Notes

• The run-result and comparison schemas are unchanged (1.0); v0.1 results and configs work as before.
• The pull-request comment option of the action has not yet been exercised on a real pull request.

Full changelog: CHANGELOG.md

v0.1.1

A correctness release.
If you gate on 0/1 metrics (success flags, collision indicators, physim.failure_rate), upgrade: v0.1.0 could report "no regression" when the candidate was clearly worse.

What changed

Statistics (verdicts can change).
Metrics whose values are all 0 or 1 now use Newcombe score intervals (hybrid score for unpaired data, Newcombe's paired method for matched trials) instead of a percentile bootstrap.
The bootstrap gave a zero-width interval when a run had no events, so "0 collisions in 30 trials" was treated as a rate known to be zero.
In simulation with 30 trials per run, a true collision-rate increase from 1% to 6% (margin 2 points) was called PASS in 15% of runs with v0.1.0, and in 0% with v0.1.1.
Expect more INCONCLUSIVE results for rare events with few trials; that is the honest answer for so little data.
Measured error rates are published in docs/comparison-semantics.md and reproducible with scripts/error_rate_study.py.

Exit codes and inputs.

• Directory inputs are hashed instead of crashing with exit 4.
• compare ... | head keeps the verdict's exit code (it used to exit 1, "regression").
• Integers too large for a float are rejected with a clear schema error.
• validate exits 2 for unreadable input, as documented.
• init accepts adapter options (-O key=value).

Packaging.
Dependency floors are now ones that actually work and are tested in CI: typer>=0.15.4, pyyaml>=6.0.1.

Docs.
New docs/oss-readiness-review.md with prioritized findings, corrected competitor comparisons (verified against their source code), and an honest positioning against scipy, statsmodels, rliable, asv and Bencher.

Install

pip install "physim-gate @ git+https://github.com/kiyeonjeon21/physim-gate@v0.1.1"
physim-gate demo

Changes

Bug fixes

• stats: Use Newcombe score intervals for 0/1 metrics
• cli: Keep exit codes correct for edge-case inputs

Documentation

• Add OSS readiness review and correct overstated claims

Build and CI

• Raise dependency floors to tested versions

Checksums (SHA-256)

15cfefd7770009b12943a80c5522ac0aaa1e995ef3e420cea472ac213fec2b4a  physim_gate-0.1.1-py3-none-any.whl
95c29e80e3e1de5f1b73cf151af56a140f281e18fc1d25c541dcfbe6b234d1d6  physim_gate-0.1.1.tar.gz

v0.1.0

First release of PhySim Gate: a simulator-agnostic regression gate for robotics, embodied AI, autonomous driving and aerial simulation results.

Given a baseline run, a candidate run and a gate.yaml, it decides metric by metric whether the candidate behaves worse than allowed, explains why, writes JSON, JUnit XML and a self-contained HTML report, and exits with a code CI can act on.

Install

Not on PyPI yet. Install from this tag, or from the wheel attached below:

pip install "physim-gate @ git+https://github.com/kiyeonjeon21/physim-gate@v0.1.0"
physim-gate demo

No GPU, simulator or container is needed.

Highlights

• Run-result schema 1.0 with explicit missing data, provenance and JSON Schemas.
• Paired and unpaired bootstrap intervals of the difference, with separate numerical, evidence and policy layers and an explicit INCONCLUSIVE outcome.
• Absolute, relative and combined margins, zero-baseline handling, a derived physim.failure_rate metric, and conservative defaults.
• CLI: init, validate, compare, report, demo, adapters, schema, with documented exit codes (0 pass, 1 regression, 2 input error, 3 inconclusive or invalid, 4 internal error).
• Adapters: json, synthetic, optional gymnasium (CPU); third-party adapters via entry points (see examples/csv-adapter).

Known limitations

Bootstrap intervals are approximate with few trials and rare events, metrics are per-trial scalars only, and LeRobot, CARLA, ROS 2 and PX4 adapters are not implemented yet. See the README and docs/roadmap.md.

Changes

Features

• Core schema, comparison engine, adapters, reports and CLI

Bug fixes

• report: Tidy terminal and HTML output

Documentation

• README, architecture, semantics, schema, adapters and roadmap

Build and CI

• Move workflow actions to Node 24 releases

Checksums (SHA-256)

0a3b61e9f3ee1baa271f9ea2907a7499cdca3ee773829589fe282efd72304367  physim_gate-0.1.0-py3-none-any.whl
867c66556f464c477d39b8c335a54965d32819245733e842b786c8348888b793  physim_gate-0.1.0.tar.gz