Skip to content

Source Based Dataset Layout

Peter Pak edited this page Jul 15, 2026 · 6 revisions

Source-Based Dataset Layout

Design agreed 2026-07-15 — not yet implemented. Effective from build 47; no retroactive migration of builds 1–46 except where noted.

Principle

Every HF dataset repo is self-contained and follows one convention:

datasets/<repo>/
  source/    raw feed data, committed to HF (the archive)
  scripts/   processing: source/ → data/
  data/      published, analysis-ready tables (parquet/jsonl)

Database, ASTM, and Conversations already work this way. This design promotes the recorder's output (the one holdout) to the same pattern and retires data/exports/ as a staging layer. Generated data has one home: the owning dataset's source/ tree.

Write path (unchanged during prints)

The print-time rule stays exactly as deployed 2026-07-01: live capture goes to the NVMe SSD, never to a git tree or the SMR HDD's random-write path. Telemetry/position/frame-metadata spool to ~/.agentic-sls/spool/; frames buffer locally. Post-print, a delivery step moves everything to its final home. The recorder never depends on submodule checkout state — if the dataset submodule is absent, delivery fails loudly and the buffer stays put.

Post-print delivery (new pipeline step)

Runs after the spool import completes, per build:

  1. Raw ticks — spool NDJSON (telemetry, position) gzipped into Inova-Mk1-Telemetry/source/spool/<build>/. Replaces the current ".imported files accumulate on the NVMe until cleaned manually" TODO — delivery IS the cleanup.
  2. Frames — packed (per-build tar/webdataset, NOT loose JPEGs — HF and git degrade past ~100k files) into the frames home (see open decision), frames.path rows updated, Replay serves from the new location. data/frames/ becomes a transient buffer, not a permanent home.
  3. Recorder metadata — builds/events export retargeted from data/exports/ into Inova-Mk1-Telemetry/source/recorder/.
  4. Processing — the dataset repo's own scripts turn source/ into data/ticks/ parquet etc. (already the existing ETL, re-pointed from data/exports/ to its own source/).
  5. Commit — one dataset commit per build is the archival act.

Unchanged: Conversations (already source-based), Postgres (stays the agent query surface — it is derived state, not publication), sidecar CSVs (build_to_inova_session.csv, sensors.csv stay in the MAIN repo — hand-authored cross-repo linkage, authoritative there).

Corrected facts (measured 2026-07-15)

  • data/frames/ = 408 GB and is the ONLY copy of frame imagery — the Telemetry dataset today holds processed ticks (151 GB) + previews, no frames, no source/. Publishing frames is therefore not deduplication; it is the first offsite/archival copy of nearly half a TB of thesis data currently living on one SMR disk.
  • NVMe headroom: ~90 GB free — comfortable for per-build spool + frame buffer (~9 GB/build average), but check before unusually long builds.
  • data/ end state: postgres/ + transient buffers only; exports/ retired after migration.

Open decisions

  1. Frames repo: pack into Inova-Mk1-Telemetry/source/frames/ (one repo, but pushes it toward TB scale) vs a new Inova-Mk1-Frames dataset (keeps Telemetry lean; one more submodule). Check HF storage guidance (~300 GB+ public repos are "contact us" territory) before choosing.
  2. Packing format: tar-per-build vs webdataset vs parquet-with-binary. Decide alongside #1; previews convention (zero-padded build dirs) should carry over.
  3. Old-era backfill: builds 1–46 frames (408 GB) and the old-era telemetry parquet in data/exports/telemetry/ should eventually move into source/ too (they are single-copy archives) — as a scheduled bulk upload, separate from the per-build flow. sls-summarize-builds re-points to the new locations when that happens.

Migration order

  1. Decide frames repo + packing (open decisions above).
  2. Build the delivery step; run it for build 47 onward.
  3. Verify one full cycle (print → spool → import → deliver → process → commit → push), then retire data/exports/ and re-point consumers.
  4. Bulk-backfill the historical archives; only after HF push verification, delete the local duplicates.

Clone this wiki locally