[Bug] `DurableStateActor.persist` never refreshes `_record` after a `DurableStateConcurrencyError`, so every later persist replays the same stale revision and the actor is wedged for its lifetime

[pathosDev]

Problem

DurableStateActor.persist computes expected from this.revision, which reads this._record?.revision. _record is written in exactly two places: preStart, from the initial load, and the success path of persist. On the failure path it is not touched — the catch block tests for DurableStateConcurrencyError, rethrows, and then rethrows again for everything else, which makes the whole block a no-op with two identical branches.

So after the first conflict the actor is stuck at a revision the store has moved past. The next command computes the same stale expected, the store rejects it with the same error, and nothing in the loop ever refreshes. Every subsequent persist fails identically, forever. The actor is not crashed — it keeps accepting commands, keeps serving a stale this.state, and keeps throwing out of the handler — so from the outside it looks alive and is permanently unable to write. Only an explicit stop-and-respawn (which re-runs preStart and reloads) recovers it.

This is not an exotic path. DurableStateConcurrencyError is the designed outcome of a second writer, and a second writer is routine: a shard rebalance that starts the entity on the new node before the old one has stopped, a false-positive downing that leaves both copies live, a projection or admin tool writing the same record, a retried request handled by a fresh actor. The store contract says "concurrent writers receive DurableStateConcurrencyError" — it does not say the loser is out of service until someone restarts it.

The object-storage backend makes the same trip and lands in the same place from the other direction. Its upsert drops the cached etag on conflict specifically so that "the next attempt fetches the real etag instead" (lines 226-236) — a fix for exactly this wedged-retry shape at the store layer. It does not help, because the actor still supplies the stale expectedRevision: the refresh path re-loads, finds the bucket's revision does not match what the actor expects, and throws again.

Evidence

The catch block whose two branches are the same statement:

src/persistence/DurableStateActor.ts:94-125
  /** Persist the new state atomically; rejects on concurrency conflict. */
  protected async persist(next: S): Promise<DurableStateRecord<S>> {
    const expected = this.revision;
    const adapter = this.stateAdapter();
    const wire = adapter ? encodeState(next, adapter) : next;
    // Store sees an envelope (or raw value when no adapter).  We re-stamp
    // the local record with the original `next` so callers see the
    // current-version domain shape.
    const upsertPromise = this.options.store.upsert<unknown>(
      this.options.persistenceId,
      expected,
      wire,
      this.persistenceOptions(),
    );
    this._persisting = upsertPromise.then(() => undefined, () => undefined);
    try {
      const record = await upsertPromise;
      const local: DurableStateRecord<S> = {
        persistenceId: record.persistenceId,
        revision: record.revision,
        state: next,
        timestamp: record.timestamp,
      };
      this._record = local;
      return local;
    } catch (err) {
      if (err instanceof DurableStateConcurrencyError) throw err;
      throw err;
    } finally {
      this._persisting = null;
    }
  }

expected comes from _record, which the catch never refreshes:

src/persistence/DurableStateActor.ts:44-46
  protected get revision(): number {
    return this._record?.revision ?? 0;
  }

and the only reload in the class is preStart:

src/persistence/DurableStateActor.ts:70-84
  override async preStart(): Promise<void> {
    const adapter = this.stateAdapter();
    const loaded = await this.options.store.load<unknown>(
      this.options.persistenceId, this.persistenceOptions(),
    );
    const option = loaded.toNullable();
    if (!option) { this._record = null; return; }
    const decoded = decodeState<S>(option.state, adapter);
    this._record = {
      persistenceId: option.persistenceId,
      revision: option.revision,
      state: decoded,
      timestamp: option.timestamp,
    };
  }

The store-side mitigation that does not reach the actor:

src/persistence/durable-state-stores/ObjectStorageDurableStateStore.ts:225-239
    } catch (e) {
      if (e instanceof ObjectStorageConcurrencyError) {
        // Drop the cached etag: the backend just told us it is stale, and the
        // `If-Match` above is built from this cache.  Keeping it meant every
        // retry re-sent the same stale etag and was rejected again, so an
        // entry stayed wedged until something happened to call `load` (which
        // refreshes the cache) or `delete`.  Forgetting it makes the next
        // attempt fetch the real etag instead.
        this.etagCache.delete(persistenceId);
        // -1 communicates "the backend rejected us, but didn't tell us the
        // current revision; load() will fetch the truth".
        throw new DurableStateConcurrencyError(persistenceId, expectedRevision, -1);
      }
      throw e;
    }

The same reasoning — "every retry re-sent the same stale value and was rejected again, so it stayed wedged" — applies one layer up and was not applied there.

Proposal

Make a conflict refresh the actor's view before it propagates:

• In the catch, on DurableStateConcurrencyError, re-load the record into _record (so state and revision are current) and then rethrow. The caller still learns its write lost — which it must, because the state it computed was derived from stale data — but the next command starts from the truth. This is the minimum and it is about five lines.
• Consider surfacing the reconciliation as a hook rather than an implicit reload — onConflict(current: S, attempted: S): S | undefined, defaulting to "reload and rethrow" — so an actor with a mergeable state can resolve the conflict in place instead of losing the write. That is the DurableStateActor analogue of what ConflictResolver gives replicated event sourcing.
• The dead if (err instanceof DurableStateConcurrencyError) throw err; line goes either way: today it is a comment written in code, and it reads as if the conflict case were handled.

Acceptance sketch

• After a DurableStateConcurrencyError, the actor's revision and state reflect the store's current record.
• A second persist after a conflict succeeds instead of throwing the identical error.
• The conflict is still reported to the caller of the failing persist — the reload does not swallow it.
• A regression test drives conflict → persist → assert success, against InMemoryDurableStateStore and at least one CAS-based store.

Reference issues: #692 asks for a DurableStateActor.onRecoveryFailure hook — the failure mode at start, where this one is the failure mode at write, and the two would sensibly land together as the class's error-handling story. #725 is the durable-state failure in the opposite direction (a decode failure is swallowed while the loaded revision is kept, so the next write overwrites good data with an empty snapshot); both are the same root habit of leaving _record in a state the store no longer agrees with. #612 concerns replay of an older authentic body against the same CAS.

Verification status

Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: reproduced by execution. A real DurableStateActor on a live ActorSystem over InMemoryDurableStateStore; one clean persist, then an out-of-band writer moves the record, then four more commands:

persist OK -> revision=1 state={"n":1} actor.revision=1
out-of-band writer moved the store to revision=2
persist THREW DurableStateConcurrencyError: durable-state concurrency conflict on counter-1: expected rev=1 but was 2 | actor.revision still 1
persist THREW DurableStateConcurrencyError: durable-state concurrency conflict on counter-1: expected rev=1 but was 2 | actor.revision still 1
persist THREW DurableStateConcurrencyError: durable-state concurrency conflict on counter-1: expected rev=1 but was 2 | actor.revision still 1
persist THREW DurableStateConcurrencyError: durable-state concurrency conflict on counter-1: expected rev=1 but was 2 | actor.revision still 1
final stored record: {"persistenceId":"counter-1","revision":2,"state":{"n":99},...}

Byte-identical error every time, actor.revision pinned at 1 indefinitely, and the actor still happily accepting commands throughout.

Part of the production-readiness review batch — tracked in #913.