Skip to content

v0.7.1 — Dataset Integrity

Choose a tag to compare

View all tags
@omertt27 omertt27 released this 01 Aug 20:28
· 28 commits to main since this release
v0.7.1
1440843

Calibra now starts with Dataset Integrity

Before diversity, coreset selection, or any other quality question, robotics teams first need to know whether they can trust their data. That's the question this release is built around — not diversity scores, not training-cost estimates, just: can I trust this dataset?

We reorganized Calibra's workflow to answer that question first, cheaply, and up front:

$ calibra integrity dataset/

✓ Timestamp consistency
✓ Episode completeness
✓ Motion is smooth (no jerk/jitter)
⚠ Camera freeze detected
⚠ Duplicate frames detected

Integrity Score: 80/100  ·  Status: Warning

Findings are grouped into Critical / Warnings / Passed rather than led with a single number — a score is still computed, but it's a summary line, not the headline.

What calibra integrity checks

  • Timestamps & sensor sync — jitter, dropout, camera lag, action/observation alignment
  • Episode completeness — statistical detection of suspiciously short episodes
  • Duplicate frames — a dropped grab or re-emitted buffer rather than a genuinely new observation
  • Camera freeze — a sustained run of near-identical frames, the camera has stopped updating
  • Blur — episodes anomalously blurry relative to the rest of the dataset
  • Jittery / jerky motion — smoothness (LDLJ), jerk spikes, velocity discontinuities in the recorded actions themselves

That last one shipped this release specifically because a LeRobot practitioner told us directly that the issues they hit most are "the basics — wrong timestamps, jittery movements, blurred camera frames." We had timestamps and blur covered; motion jerkiness turned out to live only in the old Quality-layer audit, not in the front-door Integrity check where it belonged. It does now.

Where this fits

Integrity is the first of four questions Calibra answers, in the order practitioners actually ask them:

Integrity (can I trust this?) → Quality (is it clean?) → Coverage (is it diverse enough?) → Optimization (can I train cheaper?)

Everything in this release is the first step of that pipeline — Quality, Coverage, and Optimization are unchanged and still available via calibra audit, calibra review, and calibra prune.

Also in this release

  • LeRobot v1 image decoding (--decode-images) — duplicate/freeze/blur checks now work on LeRobot v1 datasets, not just HDF5/Isaac Lab/robomimic. (v2/v3, which store frames as encoded video, are a deliberately deferred follow-up.)
  • Hugging Face Space reorganized to check Integrity first, ahead of the Quality/Coverage score, matching the CLI and docs.
  • Documentation — new Integrity Checks page, updated README workflow table and examples, mkdocs.yml nav fixes.
  • --json output for CI gating (exit code 1 on any CRITICAL finding).

Known gap

PyPI publishing for this tag needs a trusted-publisher configuration update on our end — the GitHub release and code are current regardless; pip install calibra-robotics will pick up this version once that's sorted.

Full Changelog: https://github.com/omertt27/Calibra/blob/main/CHANGELOG.md

Assets 2
Loading