v0.8.0 — Measured training results

Calibra's predictions (calibra benchmark) were always simulated — a heuristic outcome model plus linear GPU-hour scaling. That's useful for a first pass, but not evidence. This release adds the other half: a way to record what a design partner's real training runs actually cost and achieved, and to fold those measured numbers into the benchmark report wherever they're available, so a report is never presented as a validated result when parts of it are still predictions.

Added

• calibra experiment record — logs one training run's result (GPU-hours, wall-clock time, energy, eval success rate) against the design-partner protocol's full / random / calibra conditions at a given retention percentage. Training itself runs in the partner's own pipeline; this only records the outcome. Stored locally as JSON Lines at ~/.calibra/experiments.jsonl — never synced to any network endpoint. New module calibra/experiment_log.py.
• calibra experiment list / calibra experiment report — list recorded runs, or print the full retention-curve comparison for one experiment, including the Calibra-vs-random delta at each level and which (retention%, condition) pairs the protocol still expects but haven't been recorded yet. New CLI handler calibra/experiment.py.
• calibra benchmark --sweep — runs the full design-partner retention curve (default 10/25/50/75/100%, override with --fractions) in one shot instead of a single --keep fraction.
• calibra benchmark --experiment-id ID — substitutes real measured GPU-hours / eval success rate from calibra experiment record into the benchmark report wherever a matching condition and retention level has been logged, falling back to simulated values for anything not yet measured.
• Benchmark reports now carry a status: SIMULATED (nothing measured — a prediction), PARTIAL MEASUREMENT (some conditions measured, others still simulated — not safe to report as validated), or CASE STUDY / VALIDATED (full, random, and Calibra all backed by real recorded training runs). Every number is individually tagged (measured) or (simulated).
• Compute savings are now computed from GPU-hours rather than episode-count reduction, since the two can diverge once real measured numbers are mixed in.

See docs/commands.md's calibra benchmark and calibra experiment sections for the full partner workflow.

Fixed

• compute_trajectory_entropy force-cast actions to float32 before np.histogram, which could collapse distinct values (and raise "Too many bins for data range") on large-offset, sub-millimeter-variance action columns — exactly the near-duplicate-trajectory case the function is meant to detect. Now computed in float64. Reported via a Reddit user testing on real data.

Full Changelog: v0.7.1...v0.8.0

v0.7.1 — Dataset Integrity

Calibra now starts with Dataset Integrity

Before diversity, coreset selection, or any other quality question, robotics teams first need to know whether they can trust their data. That's the question this release is built around — not diversity scores, not training-cost estimates, just: can I trust this dataset?

We reorganized Calibra's workflow to answer that question first, cheaply, and up front:

$ calibra integrity dataset/

✓ Timestamp consistency
✓ Episode completeness
✓ Motion is smooth (no jerk/jitter)
⚠ Camera freeze detected
⚠ Duplicate frames detected

Integrity Score: 80/100  ·  Status: Warning

Findings are grouped into Critical / Warnings / Passed rather than led with a single number — a score is still computed, but it's a summary line, not the headline.

What calibra integrity checks

• Timestamps & sensor sync — jitter, dropout, camera lag, action/observation alignment
• Episode completeness — statistical detection of suspiciously short episodes
• Duplicate frames — a dropped grab or re-emitted buffer rather than a genuinely new observation
• Camera freeze — a sustained run of near-identical frames, the camera has stopped updating
• Blur — episodes anomalously blurry relative to the rest of the dataset
• Jittery / jerky motion — smoothness (LDLJ), jerk spikes, velocity discontinuities in the recorded actions themselves

That last one shipped this release specifically because a LeRobot practitioner told us directly that the issues they hit most are "the basics — wrong timestamps, jittery movements, blurred camera frames." We had timestamps and blur covered; motion jerkiness turned out to live only in the old Quality-layer audit, not in the front-door Integrity check where it belonged. It does now.

Where this fits

Integrity is the first of four questions Calibra answers, in the order practitioners actually ask them:

Integrity (can I trust this?) → Quality (is it clean?) → Coverage (is it diverse enough?) → Optimization (can I train cheaper?)

Everything in this release is the first step of that pipeline — Quality, Coverage, and Optimization are unchanged and still available via calibra audit, calibra review, and calibra prune.

Also in this release

• LeRobot v1 image decoding (--decode-images) — duplicate/freeze/blur checks now work on LeRobot v1 datasets, not just HDF5/Isaac Lab/robomimic. (v2/v3, which store frames as encoded video, are a deliberately deferred follow-up.)
• Hugging Face Space reorganized to check Integrity first, ahead of the Quality/Coverage score, matching the CLI and docs.
• Documentation — new Integrity Checks page, updated README workflow table and examples, mkdocs.yml nav fixes.
• --json output for CI gating (exit code 1 on any CRITICAL finding).

Known gap

PyPI publishing for this tag needs a trusted-publisher configuration update on our end — the GitHub release and code are current regardless; pip install calibra-robotics will pick up this version once that's sorted.

Full Changelog: https://github.com/omertt27/Calibra/blob/main/CHANGELOG.md

v0.7.0 — Dataset Integrity

Calibra now starts with Dataset Integrity — before diversity, coreset selection, or any other quality question, robotics teams first need to know whether they can trust their data.

$ calibra integrity dataset/

✓ Timestamp consistency
✓ Episode completeness
⚠ Camera freeze detected
⚠ Duplicate frames detected

Status: Warning

Added

• calibra integrity — new front-door command answering "can I trust this dataset?" before quality, coverage, or optimization matter. Findings are grouped into Critical / Warnings / Passed rather than led with a single score; an Integrity Score is still computed but demoted to a summary line.
  • Timestamp consistency and sensor sync (jitter, dropout, camera lag, action/observation alignment)
  • Episode completeness (statistical short-episode detection)
  • Duplicate frame detection
  • Camera freeze detection
  • Blur detection
  • Image integrity checks (duplicate/freeze/blur) now available for LeRobot v1 datasets via the new opt-in --decode-images flag
• LeRobotReader(decode_images=True) — decodes HuggingFace Image-feature columns for LeRobot v1 datasets. Off by default (increases load time/memory); no effect on the existing v2/v3 fast path.
• Homepage and documentation reorganized around the Integrity → Quality → Coverage → Optimize workflow (new docs/integrity.md, updated README, mkdocs.yml nav, demo assets under docs/demo.tape/docs/demo_fixture.py).
• Hugging Face Space reorganized to check Dataset Integrity first, ahead of the Quality/Coverage score.

Notes

• Duplicate-frame, camera-freeze, and blur checks work out of the box on HDF5/Isaac Lab/robomimic data, and on LeRobot v1 via --decode-images. LeRobot v2/v3 (video-encoded) datasets are intentionally out of scope for this release — --decode-images prints a warning and has no effect there.

Planned next

• Vision Integrity for video-backed LeRobot datasets (v2/v3) — decode a sampled subset of frames from LeRobot's mp4-encoded v2/v3 datasets so duplicate-frame/camera-freeze/blur detection work there too.

Full Changelog: https://github.com/omertt27/Calibra/blob/main/CHANGELOG.md