feat(herd): sessions that outlive your terminal (PRD 0009)

README.md

@@ -28,6 +28,12 @@ or miss one that does. A test fails the build when it drifts.
 |---|---|---|
 | `moshcode agents` | engines | list engines or launch one autonomously |
 | `moshcode start` | engines | launch an engine with its native defaults |
+| `moshcode herd` | runtime | run agent sessions that outlive this terminal |
+| `moshcode ps` | runtime | list herd sessions and what each one is doing |
+| `moshcode attach` | runtime | attach this terminal to a herd session |
+| `moshcode kill` | runtime | end a herd session |
+| `moshcode wait` | runtime | block until a session is blocked, done, or idle |
+| `moshcode restore` | runtime | rebuild the herd's sessions after a reboot |
 | `moshcode install` | engines | install an engine or workflow tool |
 | `moshcode uninstall` <br>`remove` | engines | take an engine or workflow tool off this machine |
 | `moshcode upgrade` <br>`update` | engines | update moshcode, engines, or tools |
@@ -102,6 +108,103 @@ is shorthand for `moshcode start claude`. In the TUI, use `/agents <engine>` for
 autonomous mode or `/start <engine>` for raw mode. Running `moshcode agents` or
 `/agents` without an engine still lists engines and their install status.
 
+## The herd — sessions that outlive your terminal
+
+Every launch above hands an engine the whole terminal and waits. That is why
+they feel native, and it is also why the pit can only do one thing at a time and
+why closing the lid kills the work.
+
+The herd inverts it. Add `-d` and the session runs in a runtime that outlives
+the pit, so you get your prompt back immediately:
+
+```sh
+moshcode start claude -d --name api      # runs in the background, prompt returns
+moshcode agents codex -d                 # autonomous, and still detached
+moshcode ps                              # who is running, and who wants you
+moshcode attach api                      # step in; Ctrl-b d steps back out
+moshcode kill api                        # end it
+```
+
+Close the terminal, drop the SSH link, come back tomorrow — `moshcode ps` still
+answers, and `moshcode attach` puts you back inside. In the pit the same verbs
+are `/ps`, `/attach`, `/kill`, and the roster prints on the way in.
+
+### Which one needs you
+
+Every session carries a state: `working`, `blocked`, `done`, `idle`, or
+`unknown`. `blocked` means a human decision is the only thing missing.
+
+```
+  api       claude  blocked   ~/src/coinpay        12m
+  web       codex   working   ~/src/ugig.net        4m
+  audit     opencode  done    ~/src/moshpit-dns     1h
+```
+
+State comes from one authority per session, never two. An engine that reports
+through a lifecycle hook (`moshcode herd report <name> <state>`) is believed and
+its screen is not second-guessed; everything else is classified from the bottom
+of its screen. Nothing recognisable reads `unknown`, which is a safe answer —
+detection never gates a launch. Patterns that go stale can be fixed in
+`~/.moshcode/herd/rules.json` without waiting for a release.
+
+Blocked can also come and find you, using the same notification fan-out as
+`notify()`/`ask()`:
+
+```sh
+moshcode herd notify on --ask            # email/SMS/Slack/Telegram/push
+moshcode herd start claude --name watch  # then run `moshcode herd watch` in the herd
+```
+
+With `--ask`, whatever you reply is typed into the session that was waiting.
+
+### Driving it from a script or another agent
+
+There is no second API — every verb takes `--json`, and that is what a machine
+reads. `wait` exists to be branched on: exit `0` matched, `2` timed out, `3` no
+such session.
+
+```sh
+moshcode herd start claude --name api --json
+moshcode herd prompt api "port the auth routes" --wait
+moshcode herd read api --lines 40
+moshcode wait api --state blocked --timeout 1h
+```
+
+moshscript gets the same surface as values rather than exit codes, which is what
+makes fan-out practical:
+
+```js
+herdStart("claude", { name: "api" });
+herdStart("codex",  { name: "web" });
+herdPrompt("api", "port the auth routes");
+herdPrompt("web", "port the dashboard");
+await herdWait("api"); await herdWait("web");
+say(herdRead("api", { lines: 20 }));
+```
+
+### After a reboot
+
+```sh
+moshcode restore --dry-run       # what would come back
+moshcode restore --resume        # and ask each engine to reopen its conversation
+```
+
+This brings back the *shape* — the sessions, in their directories, on their
+engines. The processes are new. Work that was in flight is not still running,
+and `--resume` only reaches engines that have a resume flag of their own.
+
+### What it runs on
+
+`tmux` when the box has it: real resizing, scrollback, native attach. Without
+tmux, sessions run under `script(1)` with their input on a FIFO — they work and
+they persist, but their size is fixed when they start. With neither, launches
+stay in the foreground and say so once. moshcode does not turn a soft dependency
+into a hard one, so `-d` never fails; at worst it degrades and tells you what
+would fix it.
+
+The session manifest and every transcript are written `0600`: engine argv and
+engine output both carry secrets.
+
 ### Parallel pit tabs
 
 At the mosh prompt, `/new` opens and switches to another independent moshcode

bin/moshcode.mjs

@@ -28,6 +28,8 @@ import { createPrd, listPrds, authoringPrompt } from "../src/prd.mjs";
 import { loginAuto, whoami, logout } from "../src/auth.mjs";
 import { tui } from "../src/tui.mjs";
 import { consoleCommand } from "../src/console.mjs";
+import { herdCommand, herdStart, splitDetachArgs } from "../src/herd-cli.mjs";
+import { detectSubstrate, substrateNote } from "../src/herd.mjs";
 import { dnsCommand } from "../src/dns.mjs";
 import { templateCommand } from "../src/templates.mjs";
 import { serveCommand } from "../src/serve.mjs";
@@ -117,6 +119,23 @@ function printEngineStatus(json = false) {
 }
 
 async function launchEngine(key, engine, args, { agentMode = false } = {}) {
+  const { detach, name, rest: engineArgs } = splitDetachArgs(args);
+  if (detach) {
+    const substrate = detectSubstrate();
+    if (substrate) {
+      const code = herdStart([
+        key, ...(name ? ["--name", name] : []), ...(agentMode ? ["--agent"] : []), ...engineArgs,
+      ]);
+      if (code) process.exitCode = code;
+      if (!process.stdin.isTTY || process.env.MOSHCODE_NESTED === "1") return;
+      return tui();
+    }
+    // R2: degrade, loudly, once — and then still do the thing that was asked
+    // for. A launch that refuses because the box has no tmux would be a worse
+    // answer than a launch that works and ends with this terminal.
+    console.error(`⚠ ${substrateNote(null)}`);
+  }
+  args = engineArgs;
   if (agentMode) {
     const note = `agent mode: ${key} ${agentLaunchArgs(engine).join(" ")}`;
     console.error(engine.agentsView
@@ -314,6 +333,18 @@ async function main() {
     const [key, engine] = resolved;
     return launchEngine(key, engine, rest.slice(1));
   }
+  // The herd (PRD 0009). `herd` is the namespace; the five verbs people reach
+  // for most often are also top-level, because `moshcode ps` is what someone
+  // types when they want to know what is running and nobody should have to
+  // learn a namespace to ask that.
+  if (cmd === "herd") {
+    process.exitCode = (await herdCommand(rest)) || 0;
+    return;
+  }
+  if (["ps", "attach", "kill", "wait", "restore"].includes(cmd)) {
+    process.exitCode = (await herdCommand([cmd === "ps" ? "ps" : cmd, ...rest])) || 0;
+    return;
+  }
   if (cmd === "tools") {
     const asJson = rest.includes("--json");
     printStatus(toolStatus(), asJson);

prd/0009-persistent-agent-runtime.md

@@ -2,14 +2,14 @@
 openprd: "0.2"
 id: "0009"
 title: "Keep the herd alive — a persistent runtime, semantic agent state, and one control surface for humans and agents"
-status: Draft
+status: Accepted
 authors:
   - anthony@profullstack.com
 created: 2026-08-09
 updated: 2026-08-09
 repo: https://github.com/moshcoder/moshcode
-discussion:
-implementation:
+discussion: https://github.com/moshcoder/moshcode/pull/341
+implementation: src/herd.mjs, src/herd-state.mjs, src/herd-cli.mjs
 tags: [runtime, sessions, agents, tui, notify]
 supersedes:
 superseded-by:
@@ -203,17 +203,26 @@ entries in a new `runtime` group:
 
 | command | what it does |
 |---|---|
-| `moshcode runtime` | start / inspect / stop the background runtime |
-| `moshcode ps` | list live sessions with state |
+| `moshcode herd` | the namespace: status, start, prompt, read, send-keys, report, notify, watch, prune, stop |
+| `moshcode ps` | list sessions with state |
 | `moshcode attach <name>` | attach to a session |
 | `moshcode kill <name>` | end a session |
 | `moshcode wait <name>` | block until a state transition |
 | `moshcode restore` | rebuild sessions from the manifest |
-| `moshcode agent <verb>` | start / prompt / read / send-keys / stop |
 
-TUI equivalents follow the existing convention: `/ps`, `/attach <name>`,
-`/kill <name>`, `/restore`. `/agents <engine>` keeps its meaning and simply
-gains `--name` and a detachable session underneath it.
+The namespace is `herd`, not the `runtime` / `agent <verb>` this document first
+proposed. Two reasons, both found while building it. `agent` is already a
+registered alias of `agents` in `PIT_COMMANDS`, and a test pins
+`suggest("agent") === "agents"` — so `moshcode agent start` would have meant two
+different things depending on where it was typed. And `runtime` is what
+`src/runtime.mjs` already calls the moshscript interpreter. The five verbs
+people reach for most are top-level anyway, which is what the original table was
+really asking for: nobody should have to learn a namespace to ask what is
+running.
+
+TUI equivalents follow the existing convention: `/herd`, `/ps`, `/attach <name>`,
+`/kill <name>`, `/wait`, `/restore`. `/agents <engine>` and `/start <engine>`
+keep their meaning and simply gain `-d` / `--name`.
 
 **The pit's front door changes.** Today `moshcode` prints a banner and a prompt.
 With anything running it prints the herd first:
@@ -301,3 +310,53 @@ exactly like today. No repeated nagging, no failure.
 - **Scope.** Phases 1–3 are independently shippable and should ship that way.
   Phase 1 alone — sessions that survive the terminal — is the bulk of the value
   and does not require a single line of state detection.
+
+## Implementation Notes
+
+Written after the build, so the document and the code agree.
+
+**A second substrate, which this PRD did not ask for.** R2 promised only to
+degrade gracefully without tmux. That was not good enough: `/new` already
+required tmux and it is the wart people notice. So there are two substrates
+behind one interface — tmux when the box has it, and otherwise `script(1)` with
+the session's stdin on a FIFO, reusing the capability detection `pty.mjs`
+already does. The FIFO is opened `O_RDWR` before the spawn so the child is its
+own writer and never sees EOF when the pit exits, which is the whole trick. Its
+one real limit: nothing outside a pty can ioctl its master, so the size is fixed
+at launch (set from inside by `stty`) and a later resize does not reach it.
+`MOSHCODE_HERD=pty` forces it, which is how the fallback is tested on a box that
+has tmux.
+
+**Two bugs the survival test caught**, both of which would have shipped as
+"finished agents report `gone`". tmux's `remain-on-exit` was being set in a
+second call, and a fast command finishes before that process starts — fixed by
+making the session and its option one invocation using tmux's `;` argument. And
+the pty substrate could not tell "the agent finished" from "the box rebooted",
+since both are a dead pid — fixed by having the session's own shell record its
+exit code on the way out.
+
+**Delivered:** R1–R12 and R14. Both substrates are covered by an integration
+test that starts a session in one process, exits it, and talks to the session
+from another.
+
+**Not delivered, deliberately:**
+
+- **R7 tier-1 hook installation.** The protocol ships and works —
+  `moshcode herd report <name> <state>` takes authority, suppresses screen
+  classification entirely while it is live, and expires so a crashed agent
+  cannot read `working` forever. What is not built is auto-installing that call
+  into each engine's hook config via the `plugins.mjs` / `skills.mjs` fan-out.
+  Until then tier 1 is opt-in and tier 2 carries the roster.
+- **R13, the browser as a real client.** `console.mjs` still points ttyd at a
+  shell rather than at `moshcode attach <name>`, and `mirror.mjs` keeps its
+  documented blind spot. The runtime it would attach to now exists, so this is a
+  small follow-up rather than a design question.
+- **R15, scrollback replay.** P2 and opt-in in this document; still the right
+  call not to write engine output across a reboot by default.
+
+**Rules will rot, and that is planned for.** The shipped patterns are
+conservative and anchored to things a terminal draws — brackets, selectors, line
+anchors — never bare English words, and a test asserts that. `unknown` is
+common and safe. `~/.moshcode/herd/rules.json` lets a rotted pattern be fixed on
+the box it rots on, and a malformed entry there loses that pattern rather than
+the file.

prd/README.md

@@ -24,5 +24,5 @@ Start one with `moshcode prd "<idea>"` (TUI: `/prd`).
 | [0006](0006-help.md) | --help | Draft |
 | [0007](0007-profullstack-site-init.md) | Generate batteries-included Profullstack sites for Moshpit names | Draft |
 | [0008](0008-ticker-research-and-plugin-marketplace.md) | Bring equity research into the pit, and ship the pit's slash commands as a plugin | Draft |
-| [0009](0009-persistent-agent-runtime.md) | Keep the herd alive — a persistent runtime, semantic agent state, and one control surface for humans and agents | Draft |
+| [0009](0009-persistent-agent-runtime.md) | Keep the herd alive — a persistent runtime, semantic agent state, and one control surface for humans and agents | Accepted |
 <!-- PRD-INDEX:END -->