Design: cross-episode action-rate variance as a curation primitive

[Sagar-024]

Context

I measured action rate across all 206 demos of a single task in lerobot/pusht (25,650 frames, no model in the loop):

• span 5.88 to 17.18 (2.92x)
• coefficient of variation 20%
• 20 demos in the 5% tails whose speed, not skill, drives the variance

Live demo: https://action-rate-variance.vercel.app

Pareto's shipped per-episode reductions (peak_velocity, mean_jerk, ...) never aggregate across demos, but velocity debiasing needs exactly that cross-episode number. The hflow curation layer (per-episode measurements, cohort statistics, SQL cuts) looks like the natural home for it.

Proposal

A cross-episode variance check that emits, per episode: action rate, cohort mean and std, z-score, percentile. Outlier pruning then becomes one query:

SELECT episode_id FROM catalog WHERE action_rate_z > 1.645

This follows the evidence-not-verdicts rule in checks.py: the check records measurements, the user writes the policy in SQL.

Questions

1. Is this the right abstraction: a check in checks.py, or a catalog-level aggregate in the curation layer?
2. Any naming or unit conventions I should follow?

Happy to write a design doc or open a PR with the metric plus tests.

[ayam04]

Great direction — I did the same kind of measurement over demos and cross-episode stats are genuinely the missing piece; the per-episode measurements floor is well-built but nothing aggregates.

On your question 1 (check in checks.py vs catalog-level aggregate): a per-episode check fundamentally cannot compute a coherent cohort statistic, because each check runs against one episode with no view of the cohort. Two clean options:

1. Check emits the raw per-episode measurement (e.g. action_rate_hz plus maybe action_span_s), exactly like episode_duration does — evidence, no verdict. Then the cohort math (mean, std, z, percentile) lives in the curation layer as a reusable SQL view/macro over the episodes view, e.g. cohort_stats(measurement_key) exposing {key}_mean, {key}_std, {key}_z, {key}_percentile. Users cut with WHERE {key}_z > 1.645 — their SQL policy, and the same statistic is available for any measurement, not just action rate, with zero per-metric check code.
2. A curation-layer aggregate that computes per-episode z-scores via a window function over the whole catalog (z = (value - avg(value) OVER ()) / stddev(value) OVER ()), materialized as a view. This keeps ONE implementation instead of one per quality metric.

I'd lean toward option 1 as the primitive (raw measurement in the check) + option 2 as the reusable cohort view in the curation layer — it composes across every existing check (peak_velocity, mean_jerk, duration_s, ...) for free, rather than being action-rate-specific.

On naming/units: hflow keys use <topic-ish>/<quantity>_<unit> (e.g. /joint_states/median_dt_s). For a cross-episode quantity I'd suggest keeping the raw key as action_rate_hz (per episode) and the derived cohort keys as action_rate_hz_z / action_rate_hz_pct in the aggregate view, so the per-episode and cohort columns are unambiguous. Since action rate is per-topic-invariant in your pusht case (one action stream), no topic prefix needed, but the aggregate should accept any measurement key so it generalizes.

One edge worth documenting in the design: cohort stats are computed over whatever catalog rows are currently open (the view re-binds per open, per the episodes view), so z-scores are corpus-relative — a 5% tail in a 206-demo corpus is different from a 5% tail in a 20-demo corpus. That's a feature (you detect speed-vs-skill demos relative to your actual corpus) but should be stated.

Happy to help flesh this into a design doc or take a first pass at the cohort view + tests if useful.

[kstonekuan]

Thanks @Sagar-024 for the measured writeup (the live demo is a nice touch), and @ayam04 for the layering analysis, which I agree with. Maintainer take:

On question 1 (check vs catalog-level aggregate): the split @ayam04 described is the right one, and it falls straight out of the design tenets:

• A check emits the raw per-episode measurement (action_rate_hz), exactly like episode_duration: evidence, no verdict, no cohort awareness. Checks fundamentally see one episode, so cohort math does not belong there.
• The cohort statistics belong in the curation layer as SQL. Per tenet 3 (our code exists only where the format forces bridging or a pitfall is non-obvious), start this as a documented DuckDB window-function pattern in docs/CATALOG.md: (value - AVG(value) OVER ()) / STDDEV(value) OVER () plus PERCENT_RANK(), rather than new API surface. The pattern composes over every existing measurement (peak_velocity, mean_jerk, duration_s, and so on) for free. If it proves common enough to be a pitfall, we can graduate it to a cohort_stats view helper later.

On question 2 (naming and units): @ayam04 has it right. Use action_rate_hz for the raw per-episode key, following the existing <quantity>_<unit> convention. Derived columns like action_rate_hz_z live only in your query or the documented view.

One thing to state explicitly in the docs: cohort statistics are corpus-relative. The z-score depends on which rows the query runs over, including any WHERE clause applied before the window. That is a feature for speed-vs-skill debiasing, but the docs example should pin it down: compute the window over the same filtered cohort you intend to cut.

A PR along those lines would be very welcome: (1) the action_rate check with tests, following the episode_duration shape, and (2) a "Cohort statistics" section in docs/CATALOG.md with the window-function query and the corpus-relative caveat. No separate design doc is needed at that scope; this discussion is the design doc.

[Sagar-024]

Thanks @ayam04 and @kstonekuan for the detailed breakdown.

The split makes sense to me now: the check only records the raw per-episode number (action_rate_hz), and the cohort math (z-score, percentile) stays in SQL on the curation side. A check only sees one episode at a time, so it cannot compute cohort stats anyway.

I'll take this on. The PR will cover:

1. The action_rate check in checks.py, following the episode_duration shape.
2. A "Cohort statistics" section in docs/CATALOG.md with the window-function query and the corpus-relative caveat.

Will post the PR link here when it's ready.

[kstonekuan]

Shipped in #40: the action_rate check and the "Cohort statistics" section in docs/CATALOG.md, implemented as designed here. Thanks @Sagar-024 for carrying it from measurement to merged code, and @ayam04 for the layering analysis that shaped it. Marking this discussion resolved.