Skip to content

Operations

Peter Pak edited this page Jul 16, 2026 · 14 revisions

Operations

Deployment model, runbook sequences, and the gotchas that were learned the hard way. (The repo's CLAUDE.md is the compact agent-facing version of this page.)

Services

One systemd user unit remains (deploy/systemd/, linger enabled): agentic-recorder (:3000) — deliberately last to move (hard "never dies mid-print" requirement). The broker (:3100) moved to compose 2026-07-16 (services/harness/): its image PINS the four harness CLI versions (claude-code 2.1.187, codex 0.144.1, opencode 1.15.12; agy 1.1.1 bind-mounted) as experimental control, auth rides bind-mounted host home dirs, and all four harnesses were smoke-verified in-container before the systemd unit retired. The dashboard (:5173) moved 2026-07-16 (services/web/).

Units run tsx without watch — editing server/src or services/harness/ does NOT hot-reload. A deploy is an explicit systemctl --user restart agentic-<name> at a safe moment.

Docker compose is the backbone for infrastructure services, one folder per service under services/ (postgres, pgadmin on :5050, web on :5173), included by the root docker-compose.yml. The root file pins the compose project name: — changing it makes compose recreate the live postgres container. Host-coupled processes (recorder/broker, harness CLIs) stay systemd for now — web crossed over 2026-07-16, recorder is next; future compose citizens: vLLM for the fine-tuned model, embedding services. The gradual migration direction (agreed 2026-07-15): infra → compose first, harnesses last.

Never restart the recorder while a print is being recorded. Check /api/health/recording first; for a graceful maintenance window use POST /api/admin/stop-after-build. Logs: journalctl --user -u agentic-recorder -f.

The recorder lifecycle self-heals: startup orphan reconciliation, spool recovery (15 s delay so the job detector re-adopts an active build first), unreachable-printer finalization.

Post-print refresh sequence

  1. uv run sls-export — refresh Telemetry source/ + Database linkage CSV
  2. uv run sls-summarize-builds — rebuild build_summaries
  3. New PrintSessions? rsync ppak@inova:/home/ppak/SLS4All/PrintSessions/ into sls4all/SLS4All-Backup (submodule since 2026-07-16) + the Database repo, then uv run sls-sync-reference
  4. New builds need a sls-match-build-sessions propose/review pass (live session stamping not yet built — see Architecture)

Tests

  • Root: uv run pytest
  • plugin/: npm test + npm run typecheck + npx tsx tests/smoke.ts (needs DATABASE_URL exported)
  • Headless agent run: npx tsx services/harness/run.ts <harness> --prompt '…'

Printer-side deploys

See Inova API Plugin for the build.sh → install.sh → reboot loop. Destructive printer actions (reboot/kill/restart) are user-executed — agents recommend and verify only. One-shot firmware restarts default to sudo reboot over the firmware's soft-restart hooks.

Conventions

  • TypeScript over JavaScript for new JS-adjacent code; Python scripts run via uv run; CLI tools installed with brew.
  • Postgres: two-arg round() needs a ::numeric cast.
  • MCP knowledge tools are read-only by construction (forced read-only pool); agent_actions logging of every *_set is expected behavior.

Gotchas (hard-won)

  • Old-era telemetry (pre-2026-07-13-restore) lives only in data/exports/telemetry/*.parquet — summarize from parquet, never --source postgres for big builds.
  • Profile names can lie (a "20mJ/mm" profile containing 15) — read the JSON field.
  • Repo unit files vs installed copies drift: deploy/systemd/*.service is the source; systemd runs the COPY in ~/.config/systemd/user/ — re-copy + daemon-reload on every unit edit (bit us 2026-07-16; compose services don't have this failure mode).
  • opencode quirks: project root = git toplevel (hence the root .opencode/ shim, the one plugin file left at repo root) and it MUST get an explicit --model — the free zen default (big-pickle) errors upstream as of 2026-07-16 and no paid provider is connected.
  • Sandboxed shells can't see listening ports — ss/curl under the sandbox falsely report healthy services as down; verify health with an unsandboxed curl.
  • The docker/mount boot race (root-caused 2026-07-15). /mnt/storage2 mounts with no hard ordering against docker; on the 2026-07-09 boot docker started first, auto-created the postgres bind path on the root filesystem, and postgres initdb'd a fresh cluster there — that was the mysterious "Jul 10 DB reset." All work Jul 10–14 (46-build restore, agent tables, build_summaries) lived in that hidden shadow until the next reboot flipped back to the stale HDD datadir. Recovered 2026-07-15 by rsyncing the shadow cluster back. Guard now in place: /etc/systemd/system/docker.service.d/wait-for-storage2.conf sets RequiresMountsFor=/mnt/storage2. If Postgres ever looks inexplicably stale: check WAL file mtimes in the datadir and look for a shadow under the mountpoint (sudo mount --bind / /mnt/rootfs-peek).
  • Wedged watchers (historical, from the pre-systemd tsx watch era): an orphaned process holding a port while a childless watcher ignores file edits. Diagnose with ss -tlnp and compare the owning PID against ps. The systemd migration removed the day-to-day exposure; server/scripts/run-recorder.sh remains for dev only (flock-guarded).
  • Worktree development during prints: for non-trivial recorder changes while recording, build in a git worktree and merge at a safe moment (worktrees need node_modules + submodule symlinks).
  • Storage tiering exists for a reason: Postgres lives on an SMR HDD that cannot absorb 10 Hz inserts — see Data Plane before "simplifying" the spool away.

Clone this wiki locally