Skip to content

Source Based Dataset Layout

Peter Pak edited this page Jul 15, 2026 · 6 revisions

Source-Based Dataset Layout

Design agreed 2026-07-15 — not yet implemented. Effective from build 47; no retroactive migration of builds 1–46 except where noted.

Principle

Every HF dataset repo is self-contained and follows one convention:

datasets/<repo>/
  source/    raw feed data, committed to HF (the archive)
  scripts/   processing: source/ → data/
  data/      published, analysis-ready tables (parquet/jsonl)

Database, ASTM, and Conversations already work this way. This design promotes the recorder's output (the one holdout) to the same pattern and retires data/exports/ as a staging layer. Generated data has one home: the owning dataset's source/ tree.

Write path (unchanged during prints)

The print-time rule stays exactly as deployed 2026-07-01: live capture goes to the NVMe SSD, never to a git tree or the SMR HDD's random-write path. Telemetry/position/frame-metadata spool to ~/.agentic-sls/spool/; frames buffer locally. Post-print, a delivery step moves everything to its final home. The recorder never depends on submodule checkout state — if the dataset submodule is absent, delivery fails loudly and the buffer stays put.

Post-print delivery (new pipeline step)

Runs after the spool import completes, per build:

  1. Raw ticks — spool NDJSON (telemetry, position) gzipped into Inova-Mk1-Telemetry/source/spool/<build>/. Replaces the current ".imported files accumulate on the NVMe until cleaned manually" TODO — delivery IS the cleanup.
  2. Frames (settled 2026-07-15 — see "Frames flow" below) — store-mode zips of ≤10k frames per chunk into Inova-Mk1-Telemetry/source/frames/<build>/<kind>/chunk-NNNN.zip, plus ML-ready parquet with EMBEDDED image bytes in the dataset's data/ (~1 GB shards, HF load_dataset-streamable). frames.path rows updated to chunk-NNNN.zip::<member>; Replay serves via ranged reads (zip central directory). The live buffer is transient, not a permanent home.
  3. Recorder metadata — builds/events export retargeted from data/exports/ into Inova-Mk1-Telemetry/source/recorder/.
  4. Processing — the dataset repo's own scripts turn source/ into data/ticks/ parquet etc. (already the existing ETL, re-pointed from data/exports/ to its own source/).
  5. Commit — one dataset commit per build is the archival act.

Unchanged: Conversations (already source-based), Postgres (stays the agent query surface — it is derived state, not publication), sidecar CSVs (build_to_inova_session.csv, sensors.csv stay in the MAIN repo — hand-authored cross-repo linkage, authoritative there).

Corrected facts (measured 2026-07-15)

  • data/frames/ = 408 GB and is the ONLY copy of frame imagery — the Telemetry dataset today holds processed ticks (151 GB) + previews, no frames, no source/. Publishing frames is therefore not deduplication; it is the first offsite/archival copy of nearly half a TB of thesis data currently living on one SMR disk.
  • NVMe headroom: ~90 GB free — comfortable for per-build spool + frame buffer (~9 GB/build average), but check before unusually long builds.
  • data/ end state: postgres/ + transient buffers only; exports/ retired after migration.

Frames flow (settled 2026-07-15)

Measured reality that shaped this: builds produce ~1M loose frame files each (chamber streams at native ~20 fps; build 44 = 1,247,262 files / 38 GB; build 40 = 58 GB). Loose files are non-negotiable-impossible on HF (millions of files) and a single zip per build exceeds HF's hard 50 GB per-file limit (build 40). Hence chunking.

PRINT      frames → NVMe buffer (fast IO; ~90 GB headroom is fine —
           builds are capped ≲12 h and the buffer is cleared before the
           next print; free-space check at print start, HDD fallback)
POST-PRINT 1. verify frame count vs spool metadata
           2. store-mode zip per kind, ≤10k frames per chunk, written
              directly into source/frames/<build>/<kind>/ (no middle copy)
           3. embedded-image parquet shards (~1 GB) → dataset data/
              (frame bytes IN the parquet — the ML serving form;
              `load_dataset` streams it without touching the zips)
           4. index parquet (ts, kind, archive, member, size) +
              frames.path rows updated in Postgres
           5. verify archives (member counts + spot hashes) →
              commit + push to HF
           6. push confirmed → delete NVMe originals

Deliberate duplication: frame bytes live twice on HF (source zips = archival originals; data parquet = ML-ready). Keep zips store-mode and the parquet image column uncompressed so Xet's content-defined chunking has a chance to dedup the identical byte runs server-side.

Open decisions

  1. Old-era backfill: builds 1–46 frames (408 GB, single-copy!) and the old-era telemetry parquet in data/exports/telemetry/ move into source/ as a scheduled bulk job, separate from the per-build flow. sls-summarize-builds re-points when that happens.

Migration order

  1. Decide frames repo + packing (open decisions above).
  2. Build the delivery step; run it for build 47 onward.
  3. Verify one full cycle (print → spool → import → deliver → process → commit → push), then retire data/exports/ and re-point consumers.
  4. Bulk-backfill the historical archives; only after HF push verification, delete the local duplicates.

Clone this wiki locally