Skip to content

Operations

Peter Pak edited this page Jul 16, 2026 · 14 revisions

Operations

Deployment model, runbook sequences, and the gotchas that were learned the hard way. (The repo's CLAUDE.md is the compact agent-facing version of this page.)

Services

Two systemd user units (deploy/systemd/, installed via deploy/systemd/install.sh, linger enabled) — the dashboard moved to docker compose 2026-07-16 (agentic-sls-web, vite dev mode, services/web/):

Unit Port Role
agentic-recorder :3000 Fastify recorder (telemetry, frames, build lifecycle, registry API)
agentic-broker :3100 Harness chat broker (SSE)

Units run tsx without watch — editing server/src or harness/ does NOT hot-reload. A deploy is an explicit systemctl --user restart agentic-<name> at a safe moment.

Docker compose is the backbone for infrastructure services, one folder per service under services/ (postgres, pgadmin on :5050, web on :5173), included by the root docker-compose.yml. The root file pins the compose project name: — changing it makes compose recreate the live postgres container. Host-coupled processes (recorder/broker, harness CLIs) stay systemd for now — web crossed over 2026-07-16, recorder is next; future compose citizens: vLLM for the fine-tuned model, embedding services. The gradual migration direction (agreed 2026-07-15): infra → compose first, harnesses last.

Never restart the recorder while a print is being recorded. Check /api/health/recording first; for a graceful maintenance window use POST /api/admin/stop-after-build. Logs: journalctl --user -u agentic-recorder -f.

The recorder lifecycle self-heals: startup orphan reconciliation, spool recovery (15 s delay so the job detector re-adopts an active build first), unreachable-printer finalization.

Post-print refresh sequence

  1. uv run sls-export — refresh data/exports/
  2. uv run sls-summarize-builds — rebuild build_summaries
  3. New PrintSessions? rsync ppak@inova:/home/ppak/SLS4All/PrintSessions/ into SLS4All-Backup + the Database repo, then uv run sls-sync-reference
  4. New builds need a sls-match-build-sessions propose/review pass (live session stamping not yet built — see Architecture)

Tests

  • Root: uv run pytest
  • plugin/: npm test + npm run typecheck + npx tsx tests/smoke.ts (needs DATABASE_URL exported)
  • Headless agent run: npx tsx harness/run.ts <harness> --prompt '…'

Printer-side deploys

See Inova API Plugin for the build.sh → install.sh → reboot loop. Destructive printer actions (reboot/kill/restart) are user-executed — agents recommend and verify only. One-shot firmware restarts default to sudo reboot over the firmware's soft-restart hooks.

Conventions

  • TypeScript over JavaScript for new JS-adjacent code; Python scripts run via uv run; CLI tools installed with brew.
  • Postgres: two-arg round() needs a ::numeric cast.
  • MCP knowledge tools are read-only by construction (forced read-only pool); agent_actions logging of every *_set is expected behavior.

Gotchas (hard-won)

  • Old-era telemetry (pre-2026-07-13-restore) lives only in data/exports/telemetry/*.parquet — summarize from parquet, never --source postgres for big builds.
  • Profile names can lie (a "20mJ/mm" profile containing 15) — read the JSON field.
  • Sandboxed shells can't see listening ports — ss/curl under the sandbox falsely report healthy services as down; verify health with an unsandboxed curl.
  • The docker/mount boot race (root-caused 2026-07-15). /mnt/storage2 mounts with no hard ordering against docker; on the 2026-07-09 boot docker started first, auto-created the postgres bind path on the root filesystem, and postgres initdb'd a fresh cluster there — that was the mysterious "Jul 10 DB reset." All work Jul 10–14 (46-build restore, agent tables, build_summaries) lived in that hidden shadow until the next reboot flipped back to the stale HDD datadir. Recovered 2026-07-15 by rsyncing the shadow cluster back. Guard now in place: /etc/systemd/system/docker.service.d/wait-for-storage2.conf sets RequiresMountsFor=/mnt/storage2. If Postgres ever looks inexplicably stale: check WAL file mtimes in the datadir and look for a shadow under the mountpoint (sudo mount --bind / /mnt/rootfs-peek).
  • Wedged watchers (historical, from the pre-systemd tsx watch era): an orphaned process holding a port while a childless watcher ignores file edits. Diagnose with ss -tlnp and compare the owning PID against ps. The systemd migration removed the day-to-day exposure; server/scripts/run-recorder.sh remains for dev only (flock-guarded).
  • Worktree development during prints: for non-trivial recorder changes while recording, build in a git worktree and merge at a safe moment (worktrees need node_modules + submodule symlinks).
  • Storage tiering exists for a reason: Postgres lives on an SMR HDD that cannot absorb 10 Hz inserts — see Data Plane before "simplifying" the spool away.

Clone this wiki locally