[Feature] Transactional state (ACID 2PC across grains)

[pathosDev]

Size / Priority

• Size: XL — own design phase + likely 8-12 week implementation slot.
• Reference: Orleans Transactions.

Rationale

The actor model deliberately avoids distributed transactions — they conflict with the "actor is the atomic unit" principle. But some domains genuinely need cross-actor ACID:

• Bank transfer — debit account A + credit account B; either both or neither.
• Inventory + Order — reserve item + create order; either both or neither.
• Multi-resource workflow — book hotel + flight + car as a unit.

Today's options:

• Saga / compensation ([Feature] Saga / Process Manager with compensation #179) — eventual consistency with explicit compensating actions on failure. Works but is verbose and requires careful retry-idempotency design.
• Single actor owns all state — collapses A + B into one actor. Limits concurrency, doesn't scale.
• External transaction coordinator (e.g., a database transaction) — bypasses the actor model.

Orleans Transactions add a fourth option: opt-in 2-phase commit (2PC) across actors, with the coordinator + transaction-log built into the framework. Use only when you really need ACID; otherwise use the saga.

This is the most ambitious item in the catalog — a major addition that touches state-store contracts, message-dispatch, and supervision. Hence XL + dedicated design phase.

Reference: what Orleans does

public interface IAccountGrain : IGrainWithGuidKey
{
    [Transaction(TransactionOption.Join)]
    Task Debit(decimal amount);

    [Transaction(TransactionOption.Join)]
    Task Credit(decimal amount);

    [Transaction(TransactionOption.CreateOrJoin)]
    Task<decimal> GetBalance();
}

public class AccountGrain : Grain, IAccountGrain
{
    private readonly ITransactionalState<Balance> balance;

    public Task Debit(decimal amount) {
        return balance.PerformUpdate(b => {
            if (b.Value < amount) throw new InsufficientFundsException();
            b.Value -= amount;
        });
    }
    // ... Credit similar ...
}

// Caller:
[Transaction(TransactionOption.Create)]
public class TransferGrain : Grain, ITransferGrain {
    public async Task Transfer(Guid from, Guid to, decimal amount) {
        await accountFrom.Debit(amount);
        await accountTo.Credit(amount);
        // 2PC commit at end of this method; debit+credit either both apply or both rollback.
    }
}

Orleans's runtime:

1. Generates a TransactionId on Transaction(Create).
2. Propagates it through every call inside the transaction scope.
3. Each touched grain joins the transaction; their state-store records the "tentative" change.
4. On scope-exit: coordinator runs Prepare phase (all participants vote), then Commit (or Abort if any voted no).
5. Failure recovery: coordinator can resume after crash via the transaction log.

Design sketch — actor-ts equivalent (high-level only)

This is a sketch of the SHAPE, not a complete design. The implementation needs its own multi-page design doc.

// src/transactions/TransactionalState.ts (new)

export interface TransactionalState<T> {
  /**
   * Read the current value inside the active transaction.
   * Throws if no transaction is active.
   */
  read(): Promise<T>;

  /**
   * Stage an update inside the active transaction.
   * Throws if no transaction is active.
   */
  update(updater: (current: T) => T): Promise<void>;
}

export interface TransactionScope {
  /** Run `fn` inside a transaction; commit on resolve, abort on reject. */
  run<R>(fn: () => Promise<R>): Promise<R>;
}

// Cluster-singleton coordinator:
export class TransactionCoordinator {
  /** Begin a new transaction; returns a TransactionScope. */
  begin(options?: { readonly timeoutMs?: number }): TransactionScope;
}

Caller:

const tx = system.extension(TransactionCoordinatorId);
await tx.begin().run(async () => {
  await accountFrom.tell(new Debit(amount));   // join tx automatically (via AsyncLocalStorage)
  await accountTo.tell(new Credit(amount));    // join tx
  // On scope exit: Prepare → Commit
});

Actor:

class AccountActor extends PersistentActor<...> {
  private readonly balance: TransactionalState<number>;

  override async preStart() {
    this.balance = await this.system.extension(TransactionalStateFactoryId)
      .create<number>(this.persistenceId, 'balance', { initial: 0 });
  }

  override async onCommand(state: never, cmd: Debit | Credit) {
    if (cmd instanceof Debit) {
      const cur = await this.balance.read();
      if (cur < cmd.amount) throw new InsufficientFundsError();
      await this.balance.update(b => b - cmd.amount);
    } else {
      await this.balance.update(b => b + cmd.amount);
    }
  }
}

Required infrastructure (huge)

• Transaction-log store — durable per-cluster log of TX states (open / preparing / committed / aborted).
• TX-context propagation — every tell / ask carries the active TX id (via AsyncLocalStorage + envelope field).
• Per-actor TX participation — actor's state-store maintains tentative + committed versions; on Prepare vote, lock the actor (block non-tx messages); on Commit, atomicify.
• Coordinator failover — singleton with recovery: new coordinator reads TX log, drives undecided TXs to completion.
• Deadlock detection — TX A holds actor X, waits on Y; TX B holds Y, waits on X. Detect + abort one.
• Timeout & abort — each TX has a deadline; coordinator aborts expired TXs.
• Read isolation — what does a non-TX read see during another TX's tentative writes? Repeatable-read? Read-committed? Snapshot? Recommend snapshot.

Out of scope / non-goals

• Cross-cluster transactions — local cluster only.
• Transactions over external resources (DB, message broker) — only actor state.
• Nested transactions — phase 1: one TX at a time per call chain. Phase 2: nested with savepoints (maybe).
• Optimistic concurrency — Orleans uses pessimistic locking. Same here for phase 1.
• Inferring transactions from method signatures — no decorators in TS; explicit tx.run(...) only.

Open design questions (many)

1. Locking granularity: per-actor (sketch) vs per-TransactionalState. Per-actor is simpler + matches actor-as-unit. Recommend per-actor.
2. Isolation level: Read-Committed vs Snapshot vs Serializable. Recommend Snapshot (best practical concurrency).
3. Coordinator placement: cluster singleton vs distributed (Raft-style). Singleton simpler; distributed scales better. Recommend singleton with persistence-based failover.
4. TX propagation: AsyncLocalStorage (sketch) vs explicit TX id parameter on every tell. ALS is cleaner; loses TS type-tracking. Recommend ALS.
5. Failure handling: actor crashes mid-TX → abort the whole TX (sketch) vs let the supervisor decide. Recommend abort.
6. Compensation vs rollback: 2PC rolls back tentative state; users with side effects (e.g. sent an email) need to handle compensation themselves. Document explicitly.
7. Phase-1 simplifications: in-memory TX log? No deadlock detection? Just timeouts? Defer features.
8. Performance: 2PC is slow (typically 2-3 cluster round-trips per TX). Document expected throughput (~100-1000 TX/sec, not 100K).

Test plan (sketch)

1. Bank transfer happy path — debit + credit; both commit; balances reflect.
2. Insufficient funds rollback — debit throws InsufficientFunds; credit also rolls back.
3. Coordinator crash mid-prepare — kill coordinator; new coordinator finishes pending TXs from log.
4. Actor crash mid-transaction — actor crashes; supervisor restarts; TX aborts; tentative changes lost.
5. Timeout — TX runs past deadline; coordinator aborts.
6. Deadlock detection — orchestrate two TXs that hold each other; detect + abort one.
7. Isolation: snapshot reads — TX A's tentative write isn't visible to TX B (or non-TX reads) until A commits.
8. Concurrent non-conflicting TXs — TX A on actors X+Y; TX B on actors W+Z; both succeed independently.
9. Saga vs Transaction comparison — same scenario implemented as Saga ([Feature] Saga / Process Manager with compensation #179) and as Transaction; behaviour should match for the happy path.
10. Performance baseline — 1K TXs/sec sustained; latency distribution documented.

Acceptance criteria (rough)

• TransactionalState<T> interface + implementations for journal/snapshot/object-storage backends.
• TransactionScope + TransactionCoordinator.
• AsyncLocalStorage-based TX-context propagation through tells.
• 2PC Prepare/Commit/Abort message flow over cluster transport.
• Coordinator singleton with failover via transaction log.
• Snapshot isolation for non-tx reads during tentative writes.
• Deadlock detection + abort.
• Documentation: "When to use Transactions vs Sagas vs single-actor-state" decision tree.
• Performance documentation.
• Test suite covers all listed cases.
• CHANGELOG entry under "New: ACID transactions across actors (Orleans-style)".

Pre-implementation checklist (mandatory)

• Multi-week joint design phase before coding starts.
• Resolve all 8 open design questions.
• Phase-1 scope cut: what's deferred to phase 2/3.
• Coordinate with [Feature] Saga / Process Manager with compensation #179 (Saga) so naming/messages don't collide.
• Coordinate with [Security] Master-key rotation: re-encryption sweep race + no progress tracking #109 (Master-key rotation) — TX log encryption.
• Reference-app + benchmark plan agreed.
• Multi-month implementation budget agreed.