feat(server): retire pm2 from operator path — launchd/systemd-user supervision + rotating logs

[aterrylu]

Problem

Terry: "pm2 has been working but also very painful — I'd like something more mainstream and reliable that works for both mac and linux."

A 10-tool market study (Tailscale, Ollama, Caddy, code-server, Supabase, Syncthing, n8n + the agent-platform peers OpenClaw/Hermes/CMUX) found a unanimous norm: tools that ship a local daemon supervise it with the OS-native init system (systemd on Linux, launchd on macOS) — none use a Node process manager (pm2/forever). autonomOS was already on-norm at both ends — autonomos install-service (shipped #170) writes a launchd LaunchAgent / systemd-user unit, and make dev runs foreground. The only off-norm leftover was pm2, surviving in the operator surface: ecosystem.config.cjs, the Makefile, and the remote-deploy SSH path. Since almost every existing user installs via make prod/make deploy, that's exactly where the migration has to live.

Solution (ADR-050, Option B)

make prod / make deploy now supervise the from-source server through the same OS-native unit the vended install uses — no pm2.

graph TD
    subgraph OLD["❌ before — pm2"]
        A1["make prod"] --> A2["bun add -g pm2"]
        A2 --> A3["pm2 start ecosystem.config.cjs"]
        A3 --> A4["pm2 daemon :3100"]
        A4 -.->|"pm2 logs / restart / stop"| A5["pm2 CLI"]
        A4 -.->|"logs append forever, no rotation"| A6["/tmp/autonomos.log ♾️"]
    end
    subgraph NEW["✅ after — launchd / systemd-user"]
        B1["make prod"] --> B2["scripts/install-prod-service.sh"]
        B2 -->|"pm2 detected?"| B3["migrate-from-pm2 (preserve PORT)"]
        B2 --> B4["generate tsx wrapper (.autonomos-bin/)"]
        B4 --> B5["autonomos install-service --bin=wrapper"]
        B5 --> B6["launchd LaunchAgent / systemd-user unit"]
        B6 --> B7["daemon :3100 (KeepAlive / Restart=always)"]
        B7 -.->|"autonomos logs / restart / stop"| B8["autonomos CLI"]
        B7 -.->|"server-owned, size-rotating"| B9["~/.autonomos/logs/autonomos.log 🔁"]
    end
    OLD ==>|"auto-migrate on next make prod/deploy"| NEW

Loading

What changed

• scripts/install-prod-service.sh replaces pm2 start ecosystem.config.cjs. Generates a tiny gitignored wrapper (exec tsx --env-file=.env packages/cli/src/index.ts "$@" — re-establishes the tsx loader + .env like make dev) and points install-service --bin=<wrapper> at it. Idempotent (re-runs reinstall + restart so new source goes live) and synchronous with a post-boot smoke check.
• pm2 auto-migration (default ON): detects a pm2-managed autonomos → migrate-from-pm2 (stop + deregister, preserve PORT) with a loud announcement, before installing the unit. NO_MIGRATE=1 escape hatch. install.sh already auto-migrates curl installs, so all paths converge.
• New CLI: autonomos logs (-f / --lines) and autonomos restart. autonomos stop is now service-aware — under KeepAlive / Restart=always a bare SIGTERM just gets revived, so stop tells the supervisor (launchctl bootout / systemctl --user stop) and restart uses kickstart -k with a bootstrap fallback (shared lib/service-control.ts).
• Server-owned rotating log (server/src/logger.ts): launchd StandardOutPath / systemd append: make the supervisor hold the log fd (unbounded, un-rotatable, two-writer-corrupting). So the server owns its log — tees stdout+stderr into a size-rotating ~/.autonomos/logs/autonomos.log (sync fd appends; keep newest N). Echo is asymmetric: stdout always (carries AUTONOMOS_READY IPC + --print-url; its supervisor sink is /dev/null), stderr only on a TTY (so the boot.error.log backstop captures just the pre-attach window and stays bounded). autonomos logs tails the rotating file.
• ecosystem.config.cjs deleted; make deploy no longer installs pm2; the systemd-user-over-ssh XDG_RUNTIME_DIR gotcha is handled.

Testing

• Unit: rotating-writer rotation/retention/failure-tolerance + the /dev/null template redirect. Full make check green — 612 server/cli + 227 dashboard tests.
• QA (isolated config dir + port 3199, never touched :3100):
  • Stage 1: wrapper→server boot, server-owned log, status/logs/stop.
  • Stage 2: real-launchd cycle — install → KeepAlive-supervised → stop stayed down (no revive) → restart (bootstrap fallback) back up → uninstall clean. System verified clean afterward.
  • Verified the asymmetric-echo fix: under a non-TTY sink, stderr lands in autonomos.log via the tee but not in the supervisor's stderr file (0 bytes) → boot.error.log stays bounded.

Risks

• make restart no longer rebuilds (it cycles the supervisor; use make prod to pick up new source) — matches pm2-restart semantics, intentional.
• First make deploy after this migrates forge off pm2 (preserving PORT=3100); subsequent deploys are clean rebuild-and-restart.

Notes

• Stacked: this is PR1 of 2. PR2 (install-UX: post-install smoke test surfaced to curl installs, URL/token print, browser/PWA auto-open) stacks on top.
• Part of the server-first direction (the Electron desktop app is being cut separately). Phase 1C: server distribution polish (CLI + install + supervision + upgrade) #170's install-service foundation stays — this finishes the migration it started.
• ADR-050 in docs/DECISIONS.md; full research in docs/research/server-lifecycle-pm2-replacement.md.

🤖 Generated with Claude Code

[nox-0x]

Approving — the pm2 → launchd/systemd-user cutover is well-executed and the design comments do the heavy lifting of explaining the non-obvious bits (KeepAlive revival, server-owned rotating log to avoid the dual-writer / supervisor-held-fd trap, asymmetric stdout-always / stderr-TTY-only echo). Idempotent install script + smoke check + auto-migration is the right operator UX.

Two minor follow-up items (non-blocking):

• 🟢 The PR description claims migration "preserves PORT" from pm2, but install-prod-service.sh always passes --port=$PORT (default $PROD_PORT=3100) to migrate-from-pm2, which then short-circuits the detected-pm2-port branch in commands/migrate-from-pm2.ts:39-44. So a pm2 user running on a custom PORT (via .env) will be silently moved to 3100 by make prod unless they also set PROD_PORT. Either drop the unconditional --port=$PORT on the migration branch (let migrate-from-pm2 detect-and-preserve), or update the description/docs to be explicit that the shell PORT wins.

• 🟢 autonomos stop is idempotent on Linux (systemctl --user stop returns 0 if already stopped) but not on Mac (launchctl bootout fails if the agent isn't loaded → exit 1, "Failed to stop the service"). Worth normalizing the "already-stopped" exit for parity.

• 🟢 packages/server/src/logger.ts:126 — getLogFilePath() is exported but unused (the CLI's logs.ts reconstructs the path from getConfigDir()). Either wire logs.ts to it or drop it.

None of these block merge.