[Feature] Deterministic Simulation Testing (FoundationDB-style)

[pathosDev]

Size / Priority

• Size: XL — own design phase + headline differentiator.
• Reference: FoundationDB Simulation, Antithesis, TigerBeetle VOPR.

Rationale

The single biggest reliability story in distributed systems: deterministic simulation testing. FoundationDB pioneered it; TigerBeetle and Antithesis turned it into commercial offerings.

The premise: every source of non-determinism in a distributed system (clock, network delays, packet drops, partitions, crashes, parallel scheduling) is mocked + seeded. A seed reproduces the exact same run, byte-for-byte. So:

• Run 10,000 randomised scenarios overnight.
• Any failure produces a seed.
• Re-run with that seed → identical failure.
• Bisect / debug deterministically.

No JS-ecosystem actor framework has this. Building it is a serious effort:

• Virtual-time scheduler (replaces setTimeout/setInterval/Date.now).
• Mock network with seeded partition/latency/drop injection.
• Deterministic random source (seeded PRNG).
• Full instrumentation of every async-boundary (Promise resolution order).
• Test-only — does not affect production runtime.

Headline differentiator. XL. Worth doing.

Reference: what FDB / TigerBeetle do

• Single-process simulation: all "nodes" run in one process, sharing a virtual-time scheduler.
• Network = in-process queues with seeded delays/drops.
• Disk = mock with seeded corruption/dropouts.
• Clock = virtual; seeded jumps and drifts.
• Workload = seeded random operations.
• Failures = seeded crash injection at chosen percentages.
• Bisection: failing seed + halved-time runs → narrows the failure window.

Design sketch — actor-ts equivalent

Three modes: production (current behaviour), simulation (seeded virtual-time + mock network), replay (deterministic re-run from seed).

// src/simulation/Simulation.ts (new)

export interface SimulationOptions {
  readonly seed: bigint;
  readonly nodes: ReadonlyArray<NodeConfig>;
  readonly faultInjector?: FaultInjector;
  readonly maxWallClockMs?: number;
}

export class Simulation {
  static start(options: SimulationOptions): SimulationHandle;
}

export interface SimulationHandle {
  /** Advance virtual time by `ms` (or until next event, whichever first). */
  advanceTime(ms: number): Promise<void>;
  /** Run a workload for `n` operations. */
  runWorkload(workload: (ctx: SimContext) => Promise<void>, n: number): Promise<void>;
  /** Force a fault at the current virtual time. */
  injectFault(fault: Fault): void;
  /** Get the full trace for replay/debugging. */
  getTrace(): SimulationTrace;
}

export interface FaultInjector {
  /** Per-tick probability of each fault type. */
  readonly partitionProbability?: number;     // p of network partition
  readonly nodeCrashProbability?: number;     // p of any-node crash
  readonly packetDropProbability?: number;    // p of any-packet drop
  readonly clockSkewMaxMs?: number;            // bounded clock skew
}

Workload + faults driven by seeded PRNG; trace records every event for replay.

Required infrastructure (huge)

• Virtual-time scheduler — replaces setTimeout/setInterval/Date.now/performance.now. AsyncLocalStorage-injected so user code is transparent.
• Deterministic Promise scheduling — every promise resolution event ordered by virtual time + seeded ties.
• Mock network transport — full alternative to runtime/tcp; in-process queues with delay/drop/reorder.
• Mock disk — alternative to all backends; with seeded I/O delays + failures.
• Seeded PRNG — single global source; every random consumer uses it.
• Trace recording + replay — every event captured; replay deterministically reconstructs.
• Workload generator — fuzz-style random operation streams.

Out of scope / non-goals

• Real-time replay — replay is logical (deterministic), not wall-clock.
• Production simulation — pure test mode.
• Cross-process simulation — single process only.
• GUI — CLI / trace-output first; viz later.

Open design questions

Many. The whole project needs its own design doc. Examples:

1. How invasive: AsyncLocalStorage injection vs runtime wrap.
2. PRNG choice (cryptographic vs fast).
3. Trace format (binary vs JSON).
4. How to make globalThis.Date.now mockable without ESM hot-patching nightmares.
5. Performance: simulation runs much slower than wall-clock; how much overhead per event.

Test plan (sketch)

1. Seeded run with same seed → identical trace, twice.
2. Seeded run with different seeds → different traces.
3. Bisection — failing seed; halve workload; identify minimal failing scenario.
4. Real bug reproduction — write a real-world bug; verify simulation catches it.
5. Performance — simulation overhead < 10× wall-clock for typical workloads.
6. Reproducibility across Bun/Node/Deno.

Acceptance criteria (rough)

• Virtual-time scheduler.
• Mock network + disk transports.
• Seeded PRNG.
• Trace recording + replay.
• Workload + fault generators.
• Reference example: cluster-split-brain simulation.
• Documentation: "Deterministic simulation testing guide".
• Test suite.
• CHANGELOG entry under "New: Deterministic simulation testing".

Pre-implementation checklist (mandatory)

• Multi-month design phase.
• Investigate FDB / TigerBeetle implementation papers.
• Resolve all open design questions.
• Coordinate with [Feature] Replay-mutation recovery fuzzer #208 (Replay-fuzzer — simpler subset).
• Phased build: virtual-time first; mock-network; seeded PRNG; trace/replay; workload; faults.
• Budget: 6-12 months for production-ready.