feat(worker): secondmate cloud monitor, send routing, and spawn gate (R2/R3 PR 4)

[ruby-dlee]

Scope (R2R3-DESIGN.md C item 4, verbatim)

"PR 4 - fm-secondmate-cloud-monitor.sh + fm-send routing + spawn gate behind FM_SPAWN_SECONDMATE_CLOUD. O_EXCL leg-dispatch discipline (reused from fm-spawn-cloud-monitor.sh); leg 2+ requests are asserted manifest-free (a golden request-shape test - the rmtree regression guard); off-flag byte-identity golden."

Design refs: R2R3-DESIGN.md B.3 (session legs), B.4 (message lane, monitor-owned chain verification, honest latency), B.9 (gate rewrite), AMENDMENT 1. Child RELAY is PR 5 and is not here: a verified fm.secondmate-child-request/v1 renders into the pane and is otherwise not acted on.

What this adds

• bin/fm-secondmate-cloud-monitor.sh (new): the compartment's local owner. Dispatches session legs through the bounded lifecycle execute behind per-leg O_EXCL markers (the fm-spawn-cloud-monitor discipline, including the stale-claim reclaim rule); leg 1 carries the payload+account staging pair, legs 2+ carry NEITHER manifest so they hit the supervisor's verified staging-skip branch instead of the rmtree; relays fm-send envelopes via the claim-exempt message-put; polls message-collect with the summary cursor (defensive against a cursorless summary shape); renews on a verified reason=wall summary, stops on close/idle with a terminal status file, and refuses renewal past FM_SECONDMATE_TTL_HOURS (72) from first dispatch. Release is deliberately NOT performed here (PR 6 / operator work). Each leg argv carries an FM_SECONDMATE_LEG=n prefix because execute dedupes by request digest: without it, leg n+1's manifest-free request would be byte-identical to leg n's and would replay its recorded result.
• bin/fm-secondmate-cloud-monitor.py (new): the chain authority (design B.4: the monitor owns chain verification). Recomputes the whole collected outbox chain: names as content addresses of the canonical unsigned message, sequences exactly 1..N, chain_digest = sha256(prev_hex + content_sha256_hex) from a 64-zero genesis. ANY break refuses the ENTIRE mailbox (including entries before the break), retains every file, writes a sticky .chain-break marker, and the bash loop freezes relay in both directions until an operator clears it. Verified replies render into the pane; declared bundles are verified size+digest against their chained declaration and landed into the local secondmate home worktree by fast-forward only when the worktree is clean and the bundle applies; anything else keeps the bundle and reports its path (the outcome-landing doctrine).
• bin/fm-send.sh: a target whose durable meta records kind=secondmate AND placement=azure routes text into state/<id>.cloud-inbox/<seq>-<sha256>.json: one canonical, content-addressed envelope per send, text carrying the same from-firstmate marker as local secondmate sends, sequence claimed with O_EXCL. --key refuses (no composer exists). Only meta written by the gated cloud spawn selects this route, so every existing send path is byte-identical (unit-pinned).
• bin/fm-spawn.sh: the KIND=secondmate half of the cloud gate opens ONLY under FM_SPAWN_SECONDMATE_CLOUD=1 (B.9 expand/contract; recovery stays gated). Flag on: the spawn stages the compartment payload (home repo bundle + brief/charter + fm-secondmate-session.py + pi extension) and the pi auth projection, queues request --role secondmate --owner-kind primary, runs reconcile (queued stays a durable notice), records the assignment, and launches fm-secondmate-cloud-monitor.sh in the tracking pane. The spawn NEVER dispatches an execute for a compartment and writes no crewmate entrypoint. Flag off: byte-identical gate behavior with the exact pre-flag notice string (golden-tested). Compartment leg config (FM_SECONDMATE_LEG_SECONDS/POLL/IDLE/TTL_HOURS, numeric only) persists into the allowlisted cloud-env file so the closed pane environment reproduces the spawn's configuration.

The envelope-generation finding (task item 2)

The runner's inbox schema on main REFUSES unknown envelope keys: validate_closed in bin/fm-secondmate-session.py admits exactly the per-kind fields plus kind and nonce, so a first-class assignment_generation key would bounce every captain message. The delivery-fencing generation therefore rides INSIDE nonce, the one documented free-form field: fm-send writes nonce = "<assignment_generation>/<seq08d>" (which also makes repeated identical texts distinct messages), and refuses to write an envelope at all when no assignment is recorded in meta or the controller. DEFERRED, explicitly: the runner-side refusal of an envelope naming a foreign generation (the provider docstring's PR 4 contract) needs a runner change; the runner is owned by the PR 2 lane and its closed schema cannot carry the field today. Until that lands, fencing is fm-send-side stamping plus message-put's exact-assignment gate in the lifecycle CLI.

wall_seconds arithmetic

The runner exits its poll loop at leg_seconds - min(300, leg/10) and its finish-leg can spend up to 600s (git bundle) + 600s (blob PUT) after that, so the supervisor wall must be at least leg_seconds + 900. The monitor grants wall = leg_seconds + 1800 (requirement plus equal slack) and refuses leg_seconds > 19800 at startup so the wall always fits the pinned supervisor's MAX_WALL_SECONDS = 21600. Default: 14400 + 1800 = 16200.

Tests (tests/fm-secondmate-cloud-monitor.test.sh, 15 units, hermetic)

The lifecycle CLI is a fixture (argv captured as unit-separator-joined lines, canned JSON, blob transfers modeled against a local store); every outbox the monitor verifies is produced by the REAL bin/fm-secondmate-session.py against that store, so the real producer feeds the real verifier. The spawn units drive the REAL fm-spawn.sh against a REAL fm-home-seed secondmate home, the fake herdr/tmux backends, and the hermetic provider protocol fixture (extended with the role=secondmate compartment tag set).

ok - leg 1 carries the staging pair; leg 2 argv is byte-exact manifest-free (rmtree guard)
ok - two racing monitors dispatch exactly one leg execute (O_EXCL)
ok - inbox relay is content-addressed and replay-safe across monitor restarts
ok - a verified chain delivers replies to the pane and close ends the chain without renewal
ok - a dropped middle blob refuses the whole mailbox, retains files, freezes relay both ways, and is sticky
ok - tampered message content refuses the whole mailbox
ok - a verified declared bundle fast-forwards a clean home worktree to the compartment tip
ok - a dirty home worktree keeps the bundle and reports its path instead of landing
ok - renewal past the TTL is refused with a terminal status, never dispatched
ok - leg_seconds above 19800 refuses at startup (wall = leg + 1800 must fit 21600)
ok - fm-send routes cloud secondmate text into fenced canonical inbox envelopes; --key refuses
ok - fm-send to a local secondmate stays byte-identical (marker + backend send, no inbox)
ok - fm-send refuses a cloud secondmate send that cannot be generation-fenced
ok - off-flag: azure + --secondmate keeps the exact gate notice and the byte-identical local path
ok - on-flag: --secondmate routes through role=secondmate request + monitor launch, with no spawn-side execute

The leg-2 golden is a byte-exact comparison of the full recorded execute argv (every field, in order), so ANY drift in the manifest-free shape goes red, not just a missing flag. Neighbor suites re-run green on this branch after rebasing onto merged PR 3: fm-worker-lifecycle, fm-spawn-cloud, fm-send-secondmate-marker, fm-secondmate-session, fm-secondmate-lifecycle-e2e. bin/fm-lint.sh (the CI shellcheck owner) exits 0 over all 243 scripts.

Mutation proofs (applied, proven red, reverted)

1. Chain verification skipped (helper swallows ChainBreak and delivers nothing/continues): not ok - timed out waiting for: chain-break marker (the dropped-blob unit).
2. O_EXCL dropped (plain truncate instead of set -C claim): not ok - two racing monitors dispatched 0 executes for one leg (O_EXCL guard broken); the exactly-one assertion catches any deviation from single dispatch.
3. Manifests added to leg 2+ (-eq 1 mutated to -ge 1): not ok - GOLDEN: leg 2 dispatch argv is not byte-exact manifest-free.

Honest limits

• The live Azure lane is proven only by the acceptance run (design C item 7): no test here talks to Azure, by design.
• The message-lane interface was consumed from the PR 3 branch tip bba27d3; PR 3 has since merged as faf8f0f and the merged lifecycle/provider files are byte-identical to that tip (verified by diff), so the merge-order dependency named in the plan is now moot. The monitor still parses the collect summary defensively (absent cursor/more treated as none/false).
• Runner-side foreign-generation refusal is deferred (see the envelope finding above).
• The recovery half of the gate keeps its pre-flag behavior under the flag; there is no new unit for recovery+azure+flag because the gate expression only narrows on KIND=secondmate and the off-flag golden pins the shared notice string.
• Compartment release/closeout, status projection, and the authority evidence mode are PR 6; child relay is PR 5.

Operator recipe (live compartment, for the acceptance run)

From the primary home, with the worker-lane FM_AZURE_* environment loaded and a seeded secondmate home at SUBHOME for ID:

1. FM_SPAWN_CLOUD=azure FM_SPAWN_SECONDMATE_CLOUD=1 bin/fm-spawn.sh <ID> <SUBHOME> --secondmate (optionally FM_SECONDMATE_LEG_SECONDS=14400 FM_SECONDMATE_IDLE_SECONDS=7200 FM_SECONDMATE_TTL_HOURS=72; leg cap 19800). The Herdr pane runs fm-secondmate-cloud-monitor.sh; it dispatches leg 1 itself once the queue entry is assigned.
2. If the spawn reports worker=queued: bin/fm-worker-lifecycle.sh reconcile --apply --confirm-subscription <sub> until assigned; the monitor converges on its own from there.
3. Talk to it: bin/fm-send.sh <ID> "<text>" (routes into state/.cloud-inbox and reports the envelope path; the monitor relays it and replies render in the pane). Time this round trip for the honest-latency evidence.
4. Renewal: let a leg cross its wall mid-conversation and confirm the pane shows the manifest-free leg n+1 dispatch and the conversation context survives.
5. End: bin/fm-send.sh <ID> ... is not the close path; write the close control by sending nothing and letting idle fire, or enqueue a fm.secondmate-control/v1 {"action":"close"} envelope manually into the inbox. Terminal status lands in state/.cloud-secondmate-status. Release stays operator work through the ordinary release lane (PR 6 adds the compartment evidence mode).

Size justification

2151 added lines against the 2000-line discipline: 1071 are the hermetic test suite, whose bulk is the suite-conventional self-contained fixtures (the fake herdr backend and the provider protocol fixture are duplicated from tests/fm-spawn-cloud.test.sh by the same convention that suite followed, plus a real seeded-home spawn world). The product surface itself is 850 lines across the two new monitor binaries and the two focused edits, single-concern per design C item 4; splitting the monitor from its gate/routing would ship an unreachable monitor or an untested gate.

Adversarial review round (head 7ae05cf)

1. BLOCKER fixed - the reclaim no longer reconstructs the rmtree after a resume. The per-leg O_EXCL marker now records the assignment generation it was claimed under; a stale MANIFEST-CARRYING claim (leg 1) whose recorded assignment differs from the controller's current one (a resume moved it, destroying the guest executed marker while the retained task disk keeps mid-leg commits) REFUSES to auto-redispatch, loudly naming both generations, and leaves the claim held for operator recovery. Under an unchanged assignment the reclaim still replays (digest-idempotent). Units: the exact resume scenario refuses with zero dispatches; the unchanged-assignment scenario replays exactly once with its staging pair. A legacy content-less marker fails closed the same way.
2. Durable verified tip in the chain verifier. The state file now persists {sequence, chain_digest} of the verified tip, and two checks run before ANYTHING is delivered: a store below the delivered sequence refuses (rewound outbox), and the recomputed chain at the tip's sequence must reproduce the tip digest (a wiped-and-reminted fresh-genesis chain, however self-consistent, refuses). Both attack shapes are units; both land in the sticky chain-break lane with files retained.
3. fm-send fences to the controller's CURRENT assignment (meta's spawn-time value is only the fallback for an unreadable controller), so post-resume envelopes never carry the dead generation. Unit: moved assignment -> envelope nonce carries the current one.
4. Provider docstrings amended (module + message_put in bin/fm-azure-worker-provider.py): the runner-side foreign-generation refusal is marked DEFERRED - the closed inbox schema cannot carry the field yet, the generation rides inside the nonce, and only the controller's exact-assignment gate enforces it today.
5. Absorbed notes: TTL overrun bound - the TTL is renewal-gated only, so the compute overrun bound is FM_SECONDMATE_TTL_HOURS plus one wall (leg_seconds + 1800), never an unbounded chain. Local-state fragility - losing state/.cloud-inbox/.claims/ re-anchors send sequencing at 1 (old and new envelopes coexist under distinct content addresses; nothing lost or deduped, ordering approximate) and losing the first-dispatch file re-anchors the TTL clock; recovery beyond that is the sticky chain-break marker and the reclaim refusal, never silent repair. (Duplicate identical sends staying two distinct turns remains by design, as above.)

Suite is now 20 units, all ok (5 new: reclaim refusal, reclaim replay, rewind refusal, re-genesis refusal, current-assignment fencing). Two new mutation proofs, applied -> red -> reverted: dropping the reclaim assignment comparison makes the resume scenario silently redispatch the manifest-carrying leg (refusal unit red); dropping the tip comparison makes the re-genesis chain verify and deliver attacker entries (unit red). fm-worker-lifecycle suite re-run green after the provider docstring edit; full-repo bin/fm-lint.sh rc=0 (245 scripts).

Post-review CI portability fix (head 8da508b): the reclaim staleness guard now branches epoch-mtime on uname (the bin/fm-lock-lib.sh idiom) because GNU stat -f %m succeeds with non-numeric output instead of failing, which silently disabled the reclaim path on Linux runners - the new refusal unit caught it in CI. Flagged, not fixed here: bin/fm-spawn-cloud-monitor.sh:86 carries the same latent BSD-first chain on its (CI-unexercised) crewmate reclaim path.