Skip to content

Data Plane

Peter Pak edited this page Jul 26, 2026 · 9 revisions

Data Plane

The recorder stack: how print data gets from the printer into Postgres, and why it's shaped that way. Design overview: Architecture §A.

Components

  • services/recorder/ — Node + Fastify recorder on :3000 (ESM, strict TypeScript). Subscribes to the Inova API Plugin's /state/stream WebSocket, polls the firmware's port-80 endpoints for camera/thermal/galvo images and rich status, and detects build start/end from the firmware heartbeats (500 ms cadence).
  • Postgres — docker container agentic-sls-postgres (pgvector/pgvector:pg16, defined in services/postgres/compose.yml), datadir bind-mounted at services/postgres/data/ (relocated from data/postgres 2026-07-15; each service folder owns its state) (co-located with the repo, trivially backed up, safe from docker compose down -v). A pgAdmin UI (services/pgadmin/) sits alongside on :5050. Chosen over Timescale/Influx/ClickHouse because MK1 scale (<1M rows/day) doesn't need them, pgvector covers future embeddings in the same DB, and LLMs speak SQL. Timescale remains a nondestructive CREATE EXTENSION away if ever needed.
  • Frames — camera/thermal/galvo images land in data/frames/<build_id>/ with metadata rows in the frames table.

Tiered storage (the spool)

High-rate streams never INSERT into Postgres during a print. The original design did, and Postgres on the SMR HDD (/mnt/storage2) couldn't keep up — WAL fsync stalls pushed ingestion lag to ~2.5–3 s against a 3 s health budget, causing the dashboard's RECORDING/NOT RECORDING flapping.

The fix (deployed 2026-07-01, mid-build-40):

  1. During a print, telemetry / position / plotter / frame-metadata streams append raw NDJSON to a spool on the NVMe (SPOOL_DIR, default ~/.agentic-sls/spool/<buildId>/<stream>.ndjson). Append lag: 10–50 ms.
  2. After the build closes, recorder/importer.ts bulk-imports into Postgres. Idempotent via spool_imported marker events written in the same transaction; files renamed *.ndjson.imported after commit; startup recovery is delayed 15 s so the job detector can re-adopt an active build first.
  3. Imported rows are identical to live-recorded ones — the importer reuses the recorders' own row mapping and takes timestamps from the frame payloads.

Low-rate writes (builds, events, health events) still go to Postgres live. /api/health/recording measures in-memory spool freshness, not MAX(ts) queries.

Recorded streams

Stream Source Destination Cadence
telemetry plugin WS /state/stream telemetry table 10 Hz (TELEMETRY_HZ; plugin can do 100 Hz, deliberately not raised)
position plugin WS /movement/position/stream position_hf event-driven — machine axes only move during recoat, so bursts between long silences
bedmatrix plugin WS thermal matrix JSON frame files + frames rows 1 Hz (native 6 Hz; storage-conservative)
camera/thermal/galvo firmware :80 image endpoints data/frames/ + frames rows 5/2/5 Hz
job detector firmware :80 status endpoints builds + events 500 ms

At build-open, builds.params stores the full raw firmware status payloads — the LLM picks fields later; don't over-curate at write time.

Table groups

  • Recording: builds (⊕ inova_session_id), telemetry, position_hf, frames, events. (plotter_commands was dropped 2026-07-15 — it never received data; root cause and resurrect path in Roadmap phase 5.)
  • Reference (synced copies): inova_jobs, inova_print_profiles, inova_print_sessions, astm_specimens — fix data at the origin repo, re-run uv run sls-sync-reference
  • Derived: build_summaries, build_registry (the linkage view)
  • Agent: agent_conversations → agent_sessions → agent_messages, agent_actions

Operator feedback (2026-07-16)

Human decisions are telemetry too. Four event kinds in events (all flow to the Telemetry dataset via recorder/events.jsonl):

  • operator_action — written by services/recorder/src/recorder/feedback.ts at the recorder's override-proxy choke point (recoater passes, full-recoat passes, layer overrides, setup overrides, part exclusion). Payload: action, params, print context (z2/phase/progress), the most recent defect_detection event within 15 min as the stimulus link, and for exclusions the part's transform+bounds captured at that moment. Agent *_set calls are logged separately in agent_actions by the MCP server — source distinguishes the channels.
  • build_layout — the full parts array (/printing/objects: id, name, mesh hash, 4×4 transform, bounds, exclusion state) once per build and again on every exclusion change, plus the camera/thermal calibration block (SafeWorkingArea, MainBox) so bbox → chamber-px → thermal-cell projection stays reproducible. Coordinates are stored NATIVE, never pre-transformed.
  • layer_objects — /plotter/objects snapshot on version change: which parts are rastered on the current layer and their outlines in the plotter workspace.
  • defect_feedback — operator verdicts on defect-model alerts from the Mission Control attention bar (POST /defect/feedback on the bridge), referencing the original defect_detection event id. Three-state per class: correct (real detection), ignored (seen but deliberately not acted on — records the decision without labeling either way), incorrect (clean negative). This is the Inova human-loop label channel the model repo names as the frontier — confirm/reject + the tick-aligned frames around the event ts = a training label.

Association with imagery needs no extra linkage: events carry ts, and the ticks parquet embeds tick-aligned chamber/galvo/thermal frames.

Registry linkage

The recorder mints integer build_ids; the firmware mints PrintSession UUIDs. sls-match-build-sessions maps them by timestamp-interval alignment with hand review in Agentic-SLS-Database/source/build_to_inova_session.csv (moved 2026-07-16 — lives beside the PrintSessions it annotates) (46/46 mapped). One firmware PrintSession can span multiple recorder builds (restarts/false starts). Live stamping of the session UUID at print start is the one remaining gap — until the plugin exposes it, each new build needs a matcher propose/review pass.

History note

The pre-2026-07-13 era was restored and renumbered into a single 46-build history, but old-era raw telemetry lives only in Agentic-SLS-Telemetry/source/telemetry/*.parquet (pushed to HF), not Postgres. Summarize big old builds from parquet, never --source postgres.


See also: Operations for deploy/restart rules · Datasets for the export contract

Clone this wiki locally