Repository navigation
Source Based Dataset Layout
Design agreed 2026-07-15 — not yet implemented. Effective from build 47; no retroactive migration of builds 1–46 except where noted.
Every HF dataset repo is self-contained and follows one convention:
datasets/<repo>/
source/ raw feed data, committed to HF (the archive)
scripts/ processing: source/ → data/
data/ published, analysis-ready tables (parquet/jsonl)
Database, ASTM, and Conversations already work this way. This design
promotes the recorder's output (the one holdout) to the same pattern and
retires data/exports/ as a staging layer. Generated data has one home:
the owning dataset's source/ tree.
The print-time rule stays exactly as deployed 2026-07-01: live capture
goes to the NVMe SSD, never to a git tree or the SMR HDD's random-write
path. Telemetry/position/frame-metadata spool to ~/.agentic-sls/spool/;
frames buffer locally. Post-print, a delivery step moves everything to its
final home. The recorder never depends on submodule checkout state — if the
dataset submodule is absent, delivery fails loudly and the buffer stays put.
Runs after the spool import completes, per build:
-
Raw ticks — spool NDJSON (telemetry, position) gzipped into
Inova-Mk1-Telemetry/source/spool/<build>/. Replaces the current ".imported files accumulate on the NVMe until cleaned manually" TODO — delivery IS the cleanup. -
Frames — packed (per-build tar/webdataset, NOT loose JPEGs — HF and
git degrade past ~100k files) into the frames home (see open decision),
frames.pathrows updated, Replay serves from the new location.data/frames/becomes a transient buffer, not a permanent home. -
Recorder metadata — builds/events export retargeted from
data/exports/intoInova-Mk1-Telemetry/source/recorder/. -
Processing — the dataset repo's own scripts turn
source/intodata/ticks/parquet etc. (already the existing ETL, re-pointed fromdata/exports/to its ownsource/). - Commit — one dataset commit per build is the archival act.
Unchanged: Conversations (already source-based), Postgres (stays the agent
query surface — it is derived state, not publication), sidecar CSVs
(build_to_inova_session.csv, sensors.csv stay in the MAIN repo —
hand-authored cross-repo linkage, authoritative there).
-
data/frames/= 408 GB and is the ONLY copy of frame imagery — the Telemetry dataset today holds processed ticks (151 GB) + previews, no frames, nosource/. Publishing frames is therefore not deduplication; it is the first offsite/archival copy of nearly half a TB of thesis data currently living on one SMR disk. - NVMe headroom: ~90 GB free — comfortable for per-build spool + frame buffer (~9 GB/build average), but check before unusually long builds.
-
data/end state:postgres/+ transient buffers only;exports/retired after migration.
-
Frames repo: pack into
Inova-Mk1-Telemetry/source/frames/(one repo, but pushes it toward TB scale) vs a newInova-Mk1-Framesdataset (keeps Telemetry lean; one more submodule). Check HF storage guidance (~300 GB+ public repos are "contact us" territory) before choosing. - Packing format: tar-per-build vs webdataset vs parquet-with-binary. Decide alongside #1; previews convention (zero-padded build dirs) should carry over.
-
Old-era backfill: builds 1–46 frames (408 GB) and the old-era
telemetry parquet in
data/exports/telemetry/should eventually move intosource/too (they are single-copy archives) — as a scheduled bulk upload, separate from the per-build flow.sls-summarize-buildsre-points to the new locations when that happens.
- Decide frames repo + packing (open decisions above).
- Build the delivery step; run it for build 47 onward.
- Verify one full cycle (print → spool → import → deliver → process →
commit → push), then retire
data/exports/and re-point consumers. - Bulk-backfill the historical archives; only after HF push verification, delete the local duplicates.