Production report: one month running a durable message bus + tmux worker orchestration (follow-up to #1493)

[badosanjos]

(Follow-up to #1493, which closed before I could share this — posting as a new thread so it's findable. Context: in that thread @stratofax described an ntfy-based private message bus, and I commented about our filesystem-NDJSON variant. This is the one-month production report.)
First some context on the architecture, since last time I only described the bus itself.

The architecture (5 layers):

• One orchestrator session, always on. My DA runs as a persistent Claude Code session and is the only thing that talks to me — voice-first (wake word on the phone, TTS back), single-voice doctrine: N workers behind it, all mute. Every worker result funnels through this one session before it reaches me.
• Workers are ephemeral Claude Code sessions in tmux. One task = one session, spawned by a CLI verb that also registers it on the bus and places it in a memory-capped systemd slice (spawn is denied under memory pressure — resource governance is part of orchestration, we learned that the hard way). Being tmux, I can attach to any worker mid-run, watch it think, type into it, and detach — the run continues. That attachability turned out to be the feature I'd defend last.
• The bus is just the filesystem. Append-only NDJSON: each worker gets an inbox (progress/need-input/done) and an outbox (answers/redirects/cancel); a main queue routes what needs my attention. No broker, no daemon for transport — a tail -f is a debugger.
• Delivery lives in GitHub issues, not in agents. Workers get promoted to issues on a private kanban (done → in-review → my review verb closes and reaps the session). The agent is ephemeral; the delivery record isn't. Each issue carries a recovery block: {tmux name ↔ session uuid ↔ cwd}, pinned at spawn.
• Pulse is the pane of glass. Everything above feeds the LifeOS dashboard: a live org tree of agents (who's running, blocked, or dying — drill-down is read-only by doctrine), the work kanban, and the notification stream. The bus is watched via fs events, so the dashboard updates in near-real-time without polling the agents themselves. Plus ntfy push to my phone whenever a worker finishes or needs input — the escalation queue distinguishes "FYI" from "decision needed" so my phone only interrupts me for the latter.

A month in, what held up: the durable bus survived a real VM restart with zero loss (moved off /tmp early — do this on day one). Splitting the human-facing queue into decision / FYI / parked lanes killed head-of-line blocking (one pending decision once jammed everything for 19 hours). And session pinning turned disaster recovery into routine: an OOM killed 7 concurrent sessions mid-work and every one came back with --resume <uuid>, full context.

What didn't: "ok: true" is not consumption — a worker sat blocked 3 hours because a TUI hijacked its pane while my answer-write "succeeded"; we now verify cursor movement, not delivery. Detection layers breed their own bugs (every incident spawned a detector, then a detector for the detectors) — the root cause was three state stores (bus, registry, issues) that could disagree, and the fix is making the registry a pure projection of the event log. And message durability ≠ work durability: when a parent session dies, its children are connected only by a label, not an edge. We benchmarked ~200 projects this week; conclusion was don't migrate (nothing out there supervises an interactive CLI in tmux), but steal three ideas — parent→child edges at spawn (Temporal's ParentClosePolicy, OTP's links), heartbeat-with-payload instead of poll-based detectors, and state-as-projection-of-log.

~80 workers through the bus so far, a dozen concurrent sessions at peak, one VM, voice as the main interface most of the day. Happy to go deeper on the lanes design, the session-pinning, or the Pulse wiring if useful.