Field report: HFlow on real Egocentric-10K footage (3 factories, 128 episodes)

[Sagar-024]

Ran the pipeline on real Egocentric-10K factory footage, then attacked it: repeat runs, two concurrent processes, a 600s and a 1190s episode, full 1080p, a bucket-backed root, a mid-run kill, and five corruption modes. Every bullet has the command that produced it.

Bottom line

• Worked: direct H.264 landing is 309 MB from 946 MB of source, 0.33x. Checks caught 8 of 8 injected faults across three factories, zero false flags on 120 clean episodes. 1080p and bucket-backed root run end to end. A mid-run kill leaves the catalog clean.
• Worked: concurrency scales to 4 workers at 1.95x, and memory stays linear at about 48 KB per frame.
• Did not: re-runs record four different ways by path; the input boundary refuses corrupt bytes with raw mcap tracebacks; one example hardcodes episode identity; camera-less episodes get an IndexError as diagnosis; worker scaling plateaus after 4.
• In review from this run: decayed-file re-certification ships as fix(checks): validate canonical CRCs once at lane entry with memoization #502.

What I ran

• builddotai/Egocentric-10K, revision 3e5f87c8..., three pinned shards: factory_051, factory_002, factory_012, manifest-verified hashes.
• 128 episodes: 640x360, 10 FPS, 20 s, 200 H.264 messages each; 8 with injected faults, 120 clean. Plus 600s, 1190s, and full-1080p episodes for the ceilings.
• WSL2 Ubuntu, Python 3.12, 16 cores, no GPU.
• Flow: prepare.py then pipeline.py per the example README, process_many for the worker passes, one bucket-backed pass.

Numbers, one line each

• prepare, download plus 96 episodes: 7m33s, peak RAM 673 MB
• pipeline cold: 10m37s, peak RAM 108 MB; warm: 3m05s
• 2 concurrent processes: 96 episodes in 63s, 2.9x vs warm sequential
• landing vs source: 0.33x
• 600s episode: 49.8s, peak RAM 326 MB
• 1190s episode: 1m07s, peak RAM 570 MB
• 1080p episode: 24.7s, camera_health 3.8s, peak RAM 735 MB
• bucket-backed root: end to end OK, re-process 2.3s
• curate plus export plus verify, 128 episodes: 39.3s + 71.4s + 0.1s, verify OK with 0 findings

Findings

1. Hardcoded episode IDs at prepare.py:168 and :494. The factory_002 shard lands 16 episodes named factory_051_episode_XXXX; running factory_012 next silently overwrites, 32 become 16, and survivors carry factory_051_worker_001 as operator on factory_012 footage. Fix: derive the id from the source identity.
2. Re-runs record four ways: reuse path zero new rows after 10m37s of checks; process_many doubles 30 of 32 episodes; kill mid-check zero rows zero corruption; restart appends and doubles. Cause: duration_s inside the content-addressed filename makes a re-run look new, while the reuse path skips recording. Aggregations double-count the doubles and miss the drops.
3. Input boundary: five corruption modes, five raw mcap tracebacks (zero-byte EndOfFile, half-truncated and missing-footer RecordLengthLimitExceeded, wrong-container InvalidMagic, flipped summary byte). All exit 1, none names the damage. Recoverability is clean: catalog byte-identical before and after, zero orphan rows. Contrast: the plan boundary refuses a 1204s excerpt against a 1200s source with a named numbered message. Bad plans get clean errors; bad bytes get tracebacks.
4. Decode is linear, measured at four sizes: camera_health 744 ms at 200 frames, 6181 at 2000, 22029 at 6000, 34704 at 11900; per-frame cost holds at 2.9-3.7 ms throughout. Honest correction: my notes first called this superlinear by comparing two different checks; measuring both at three sizes killed that claim before it reached this post. camera_health also does 13-30x the work of camera_frame_stats at every size, which looks like a second decode pass inside one registration. Memory law: about 48 KB per frame of peak RSS; a 3h episode crosses ~2.5 GB and a 4h episode does not fit an 8 GB laptop.
5. The checks are no longer the expensive part: camera_frame_stats is 57 ms per episode at 640x360. Contact-sheet enrichment costs more than every check combined, about 1.16 s per episode. The cold run puts roughly 70% of wall time in transform/IO with no per-step timers.
6. process_many plateaus after 4 workers: 2 workers 38.4s, 4 workers 19.7s, 8 workers 15.2s. Not CPU (8 workers peak at 259% of 1600%) and not filesystem (ext4 vs /mnt/c within 10%). The serialization sits in the coordination layer, most plausibly the per-episode ffmpeg subprocesses plus catalog writes.
7. Camera-less episode, a legitimate state-only recording: two check wrappers report raw IndexError: list index out of range in the error field while five other checks measure it fine. No crash, but the diagnosis never names the condition.
8. The README never says the dataset is gated; the first authenticated run hits Access denied. One sentence fixes it.

Scope

Tested: sync, transform, checks, enrichment, catalog, export, verify, bucket root, 640x360 and 1080p, interruption. Not tested: the Airflow scheduling shell, which needs Docker group access on this machine.

Every bullet has its command. Ask for any of them.

[Sagar-024]

@kstonekuan report is up, commands on every bullet. Three calls I want yours on before I go deeper:

1. Re-run identity: episode plus check only, duration_s out of the content address. One rule kills all four behaviors. Agree?
2. Corrupt bytes get mcap tracebacks while bad plans get named refusals. I'd give bytes the same named-refusal layer at ingest. Want the PR?
3. camera_health costs 13-30x camera_frame_stats at every size on the same frames. Second decode pass by accident, or intended?

Say which one you want first and it's mine. No rush on any of it, I know the week you're having.