Repository navigation
Operations
Deployment model, runbook sequences, and the gotchas that were learned the hard way. (The repo's CLAUDE.md is the compact agent-facing version of this page.)
One systemd user unit remains (deploy/systemd/, linger enabled):
agentic-recorder (:3000) — deliberately last to move (hard "never dies
mid-print" requirement). The broker (:3100) moved to compose 2026-07-16
(services/harness/): its image PINS the four harness CLI versions
(claude-code 2.1.187, codex 0.144.1, opencode 1.15.12; agy 1.1.1
bind-mounted) as experimental control, auth rides bind-mounted host home
dirs, and all four harnesses were smoke-verified in-container before the
systemd unit retired. The dashboard (:5173) moved 2026-07-16
(services/web/).
Units run tsx without watch — editing server/src or services/harness/ does NOT hot-reload. A deploy is an explicit systemctl --user restart agentic-<name> at a safe moment.
Docker compose is the backbone for infrastructure services, one folder per service under services/ (postgres, pgadmin on :5050, web on :5173), included by the root docker-compose.yml. The root file pins the compose project name: — changing it makes compose recreate the live postgres container. Host-coupled processes (recorder/broker, harness CLIs) stay systemd for now — web crossed over 2026-07-16, recorder is next; future compose citizens: vLLM for the fine-tuned model, embedding services. The gradual migration direction (agreed 2026-07-15): infra → compose first, harnesses last.
Never restart the recorder while a print is being recorded. Check /api/health/recording first; for a graceful maintenance window use POST /api/admin/stop-after-build. Logs: journalctl --user -u agentic-recorder -f.
The recorder lifecycle self-heals: startup orphan reconciliation, spool recovery (15 s delay so the job detector re-adopts an active build first), unreachable-printer finalization.
-
uv run sls-export— refresh Telemetrysource/+ Database linkage CSV -
uv run sls-summarize-builds— rebuildbuild_summaries - New PrintSessions? rsync
ppak@inova:/home/ppak/SLS4All/PrintSessions/intosls4all/SLS4All-Backup(submodule since 2026-07-16) + the Database repo, thenuv run sls-sync-reference - New builds need a
sls-match-build-sessionspropose/review pass (live session stamping not yet built — see Architecture)
- Root:
uv run pytest -
plugin/:npm test+npm run typecheck+npx tsx tests/smoke.ts(needsDATABASE_URLexported) - Headless agent run:
npx tsx services/harness/run.ts <harness> --prompt '…'
See Inova API Plugin for the build.sh → install.sh → reboot loop. Destructive printer actions (reboot/kill/restart) are user-executed — agents recommend and verify only. One-shot firmware restarts default to sudo reboot over the firmware's soft-restart hooks.
- TypeScript over JavaScript for new JS-adjacent code; Python scripts run via
uv run; CLI tools installed with brew. - Postgres: two-arg
round()needs a::numericcast. - MCP knowledge tools are read-only by construction (forced read-only pool);
agent_actionslogging of every*_setis expected behavior.
-
Old-era telemetry (pre-2026-07-13-restore) lives only in
data/exports/telemetry/*.parquet— summarize from parquet, never--source postgresfor big builds. - Profile names can lie (a "20mJ/mm" profile containing 15) — read the JSON field.
-
Repo unit files vs installed copies drift:
deploy/systemd/*.serviceis the source; systemd runs the COPY in~/.config/systemd/user/— re-copy + daemon-reload on every unit edit (bit us 2026-07-16; compose services don't have this failure mode). -
opencode quirks: project root = git toplevel (hence the root
.opencode/shim, the one plugin file left at repo root) and it MUST get an explicit--model— the free zen default (big-pickle) errors upstream as of 2026-07-16 and no paid provider is connected. -
Sandboxed shells can't see listening ports —
ss/curlunder the sandbox falsely report healthy services as down; verify health with an unsandboxed curl. -
The docker/mount boot race (root-caused 2026-07-15).
/mnt/storage2mounts with no hard ordering against docker; on the 2026-07-09 boot docker started first, auto-created the postgres bind path on the root filesystem, and postgres initdb'd a fresh cluster there — that was the mysterious "Jul 10 DB reset." All work Jul 10–14 (46-build restore, agent tables,build_summaries) lived in that hidden shadow until the next reboot flipped back to the stale HDD datadir. Recovered 2026-07-15 by rsyncing the shadow cluster back. Guard now in place:/etc/systemd/system/docker.service.d/wait-for-storage2.confsetsRequiresMountsFor=/mnt/storage2. If Postgres ever looks inexplicably stale: check WAL file mtimes in the datadir and look for a shadow under the mountpoint (sudo mount --bind / /mnt/rootfs-peek). -
Wedged watchers (historical, from the pre-systemd
tsx watchera): an orphaned process holding a port while a childless watcher ignores file edits. Diagnose withss -tlnpand compare the owning PID againstps. The systemd migration removed the day-to-day exposure;server/scripts/run-recorder.shremains for dev only (flock-guarded). - Worktree development during prints: for non-trivial recorder changes while recording, build in a git worktree and merge at a safe moment (worktrees need node_modules + submodule symlinks).
- Storage tiering exists for a reason: Postgres lives on an SMR HDD that cannot absorb 10 Hz inserts — see Data Plane before "simplifying" the spool away.