[Bug] v0.8.0: duplicate-snapshot byte mismatch closes a healthy worker's control channel and terminally fails the root session

[umyunsang]

Reference: follows up on #1229 (closed as not_planned during the triage migration). This issue is still present in v0.8.0; fresh reproduction below.

Summary

The bug reported in #1229 is still reproducible on v0.8.0. A snapshot cache byte mismatch ("Duplicate snapshot ... did not match cached bytes") escalates to closing the supervisor↔worker control channel of an otherwise healthy worker, which puts the resident worker through recovery, ends in terminal lifecycle: "failed" ("Waiting for a client with fresh runtime context"), and bricks the session: every subsequent open/resume of that transcript is rejected with:

Failed to open agent: Session "/Users/me/.prime/agent/sessions/<id>.jsonl" is registered to a failed worker that could not be safely reclaimed

Meanwhile the worker process itself stays alive and keeps doing real work (model calls, subagents, transcript writes), invisible and uncontrollable from any client.

I also have a concrete ordering finding for why two encodings of the same snapshotId can differ, beyond the hypotheses listed in #1229.

Environment

• Prime Agent 0.8.0 (latest release, installed 2026-08-21 via install script)
• macOS 26.5 (Apple Silicon), Node v26.5.0
• One long-running root session (~15 MB JSONL transcript, 1,492 messages, 145 RLM child spawns) plus several other resident roots

Incident timeline (all times UTC, 2026-08-23)

From the supervisor log for the affected worker:

03:11:07 Failed to catch up client ...: Error: Duplicate snapshot 9f5082ac63db-4f50e5e4aea6-15416 did not match cached bytes
03:11:18 ... Duplicate snapshot 9f5082ac63db-4f50e5e4aea6-16483 did not match cached bytes
03:13:01 ... Duplicate snapshot 9f5082ac63db-4f50e5e4aea6-31804 did not match cached bytes
        (each followed by get_connection_state / get_messages / get_session_context failing: "Session worker is recovering")
03:13:22 Recovered worker ece296ed6458 without replaying uncertain operations: ...
03:14:14 Supervisor command prompt failed: Error: Session worker is failed
03:14:47+ Supervisor command create failed: Error: Session "..." is registered to a failed worker that could not be safely reclaimed   (repeats indefinitely)

Worker descriptor afterwards:

{
  "workerId": "ece296ed6458",
  "pid": 48829,
  "rootActiveSessionId": "9f5082ac63db",
  "sessionFile": ".../01a02a6e-952f-735d-bf03-efbceef68bfc.jsonl",
  "lifecycle": "failed",
  "consecutiveFailures": 2,
  "lastError": "Waiting for a client with fresh runtime context"
}

Key observation: the worker never died. The worker's own log contains only its startup line, zero errors. While the supervisor considered the session dead:

• the transcript file kept growing (measured +20 KB over 25 s, repeatedly),
• 22+ RLM child session files were being modified within the last minute,
• but prime-agent list froze at exactly 1,492 messages across repeated calls while the file grew by hundreds of KB — i.e. the supervisor's view of the worker was frozen at the moment the channel died.

So this is purely the control/observation plane severing, then the recovery policy converting a display-layer cache problem into a terminal session failure.

Root cause: snapshot identity vs. content are captured non-atomically

snapshotId is a logical event-cursor position used as a byte-level transfer identity:

// packages/coding-agent/src/modes/daemon/daemon-mode.ts, both call sites
const snapshotId = `${state.activeSessionId}-${state.eventGeneration}-${state.lastEventSequence}`;

But the messages serialized under that ID are captured at a different time than the ID, and inconsistently so:

1. Attach path (daemon-mode.ts:3903): result comes from await this.createAttachResult(...) first; snapshotId is computed after the await. If events land between reading messages inside createSessionSnapshot and computing the ID here, the cached generation stores ID(seq=N) with content(seq=M>N).
2. Replacement path (daemon-mode.ts:6659, prepareReplacementSnapshot): the opposite order — snapshotId is computed before await this.createAttachResult(...). Same ID can now be paired with newer content on re-encode.
3. createSessionSnapshot (daemon-mode.ts:5046) awaits buildRlmChildSnapshotsWithPassiveRlmSubagents (with up to MAX_SESSION_SNAPSHOT_STABILIZATION_RETRIES = 3 stabilization retries) while the session keeps appending events, and returns messages: session.messages — a live array reference. The chunker only shallow-copies (const messages = [...options.messages], snapshot-transcript-cache.ts:34) and serializes lazily when consumed.

Any of these makes two encodings sharing (activeSessionId, eventGeneration, lastEventSequence) legitimately differ in bytes. The supervisor's duplicate validation then treats a benign capture-ordering artifact as a protocol violation:

// packages/coding-agent/src/modes/daemon/daemon-supervisor.ts:4347-4362
throw new Error(`Duplicate snapshot ${generation.transcript.snapshotId} did not match cached bytes`);
...
} catch (error) {
    this.failWorkerSnapshotCache(worker, activeSessionId, ..., true /* closeWorkerChannel */, ...);

closeWorkerChannel=true closes the private transport to the healthy worker, handleWorkerClose starts recovery, and because a resident worker has no stored launch env / transient create command, recovery cannot relaunch it and marks it terminally failed after retries ("Waiting for a client with fresh runtime context"). The stale registration then blocks every future open of that transcript until a fresh create forcibly stops the (still working) process.

Large transcripts amplify all three windows: serialization of a 15 MB history takes long enough that concurrent appends during capture become routine, which matches #1229's observation that affected sessions were the large, busy ones.

Proposed fix (staged, smallest first)

1. Contain failure to the transfer, not the worker. On a duplicate/same-ID metadata or chunk mismatch, invalidate that snapshot generation and force a fresh resync snapshot for the affected client. Do not pass closeWorkerChannel=true for snapshot-content mismatches; reserve channel closure for auth/framing/routing corruption.
2. Make (ID, content) atomic or make IDs unique. Either freeze the committed cut (capture lastEventSequence and materialize chunks under one synchronous boundary, retrying if the revision moved), or allocate a unique opaque transfer ID per materialization while keeping (eventGeneration, lastEventSequence) as the base position metadata. The latter is wire-compatible since clients treat snapshotId opaquely.
3. Optional hardening from v0.7.1: snapshot position is reused as transfer identity; byte mismatch disconnects a healthy worker #1229's design: per-transfer validation state, worker-incarnation fencing, bounded transfer lifetimes.

Step 1 alone converts this from "bricks a live session" to "client resyncs", which matches what the mismatch actually is.

Suggested regression tests

• Pause event processing after a message mutation but before the sequence advances; encode the same cursor twice; assert the worker channel survives and the client receives a resync snapshot.
• Two encodings at the same cursor with one interleaved append must never share byte-identity expectations (unique transfer IDs or frozen cuts).
• Force a same-transfer conflict; assert worker, root, and sibling sessions stay live.
• One backpressured client entering catch-up repeatedly (no second client needed), matching both v0.7.1: snapshot position is reused as transfer identity; byte mismatch disconnects a healthy worker #1229's logs and this incident.

Notes

• Happy to provide full unredacted logs, the worker descriptor, or help validate a fix against this workload (it reproduces readily: keep one large busy root session and detach/reattach a TUI client during active streaming).
• Posting this as a discussion per the contribution guidelines rather than a PR.

[umyunsang]

Update: both fix stages are implemented locally on v0.8.0 and validated against the original incident workload.

Stage 1 - containment (stop bricking healthy workers)

Patched the four fatal sites in daemon-supervisor.ts so a same-ID snapshot mismatch invalidates only that transfer (closeWorkerChannel=false) instead of closing the worker channel:

• begin frame: "did not match the cached transfer"
• begin frame: "restarted before completion"
• chunk mismatch: "Duplicate snapshot ... did not match cached bytes"
• end frame: metadata mismatch / missing result

Cache eviction and client resync behavior is unchanged - only the escalation-to-worker-recovery is removed.

Stage 2 - root cause (atomic capture of ID and content)

Made the snapshot cut atomic so an ID always maps to one byte-stable state:

1. createSessionSnapshot(): after child-snapshot stabilization, synchronously freeze [...session.messages] together with (eventGeneration, lastEventSequence) in one block with no await between the reads, and return those frozen values instead of live reads.
2. createAttachResult(): derive replay, top-level lastEventSequence, and lastEventCursor from the frozen cut instead of re-reading live state after the awaits.
3. Attach path: build snapshotId from result.snapshot.lastEventCursor instead of live state.
4. Replacement path: resolve the ID inside prepareReplacementSnapshot() after createAttachResult() returns, from the result's cursor, instead of computing it from live state before the await.

With this ordering, two encodings of the same cursor necessarily serialize the same frozen message array, so the supervisor's duplicate byte-check becomes a valid invariant rather than a latent false-positive.

Validation

Applied to the installed v0.8.0 bundle and exercised against the exact workload that produced this incident (18 MB root transcript, 1,492 messages, 145 RLM children, goal-continuation auto-resume):

• Daemon restarted on the patched bundle; supervisor + fresh resident workers run the patched code.
• Headless smoke runs pass (--no-session -p round-trip OK).
• The affected session was stopped, reclaimed via doctor --fix, and reopened cleanly with full history; kernel restored from its persistent state.
• No Duplicate snapshot ... did not match cached bytes occurrences since the restart, across active attach/catch-up cycles on the large session.
• Before the fix, the stranded control-plane left a goal continuation loop failing ~675 consecutive turns ("Kernel has been shut down"); that entire failure class disappears once the channel survives mismatches.

Caveat: this validation is operational, not yet a formal test suite. If maintainers are open to it, I'm happy to prepare a proper PR from source (not the bundle diff) with the regression tests sketched in the report plus a changelog fragment.

[tkellogg]

@umyunsang do you have code pushed somewhere? the latest v0.8.1 is basically unusable to me