Semantic mutation testing over Temporal workflows. This lab plants deliberate bugs, mechanical and AI written, in a Temporal (TypeScript SDK) payment workflow system, and judges every one by replaying recorded histories instead of running anything live: a verdict in milliseconds, with no server. On that instrument runs the headline act, an agent that treats one question, where is this workflow's test suite blind, as a budgeted learning problem: it probes the most promising places, updates on what it finds, and has every decision measured against hypotheses written down before the run.
This is a specific solution, not a general tool. The target is one payment
workflow on the Temporal TypeScript SDK, the oracle leans on Temporal's own
replay determinism, and every number below was measured here. The method and its
sharp edges are documented in docs/spikes/ so it can be ported deliberately.
Of the two hypotheses we pre registered about the agent, one was refuted and published, and the other was rescued only when an adversarial review caught its metric counting the wrong unit. That is the point: the decisions in this repo are measured, not vibes, and the record shows the misses.
Status: complete and measured. The agent is evaluated in a gap gym where the right answer is machine derived, and the live loop plus its keyless replay are reproducible from committed runs.
Two commands. The demo runs the whole lab: the instrument arms, the agent measured in its gym, and a byte for byte audit of the committed flagship agent run, ending at the scored report. No Temporal server, no Docker, no API keys.
make dev-setup # install dependencies
make demo # everything, end to end, then open site/index.htmlEach act of the demo is also its own command when you want one question answered:
make gym # is the agent's learning real? three policies vs planted blind spots
make agent-replay # can I audit a recorded agent run? replays it byte for byte
make agent # run the agent live on this code (the ONLY target that spends
# money, needs .env keys, stops at TMUTATE_LLM_BUDGET_USD)make help lists everything else, grouped the same way.
Some programs run for days: a payment waiting on a bank, an approval that times out after 24 hours. Temporal is an engine that keeps such long running programs (workflows) alive across crashes by writing every step into a diary called the event history. This project asks: if a bug crept into that long running code, would your tests notice? It plants small deliberate bugs one at a time and checks whether any test raises an alarm. The trick that makes it fast: judge each buggy version by replaying the saved diary of a correct run. The moment the buggy version tries a step the diary does not show, the alarm fires, in milliseconds, with no server. And because one experiment costs milliseconds, an agent can afford hundreds of them.
This is what the diary buys Temporal, with or without any of the mutation testing above: a workflow that survives a crash, a restart, or years of waiting, by writing down every step as it happens and reading the whole diary back before doing anything new.
flowchart LR
subgraph Before["before a crash"]
A1["authorize the card"] --> W1["written to<br/>the diary"]
W1 --> A2["capture attempt 1,<br/>the gateway times out"]
A2 --> W2["written to<br/>the diary"]
end
subgraph After["after a crash, maybe on another machine, maybe hours later"]
R["read the whole diary<br/>back first"] --> P1["authorize replayed<br/>instantly, the card is<br/>not charged a second time"]
P1 --> P2["capture attempt 1<br/>replayed instantly,<br/>not retried against the gateway"]
P2 --> S3["capture attempt 2<br/>runs for real,<br/>for the first time"]
end
W2 -.-> R
Replaying the authorization and the failed capture attempt has to land on the exact same outcome every time, or the diary and reality drift apart, and the customer could be charged twice. That is the determinism guarantee this whole project leans on: replay a diary against any version of the workflow code and a divergence means that version would have behaved differently, which is exactly what a mutation tester wants to know.
This is the actual business the target system runs: one payment moving through authorization, an optional human approval, capture with retries, and either a settlement or a compensating refund.
flowchart LR
START(["payment intent<br/>created"]) --> AUTH["authorize the card"]
AUTH -- "declined" --> CANCELED["canceled,<br/>webhook fires"]
AUTH -- "authorized" --> GATE{"above the<br/>approval threshold?"}
GATE -- "no" --> CAPTURE
GATE -- "yes" --> APPROVAL["pending approval,<br/>wait for a human"]
APPROVAL -- "approved" --> CAPTURE["capture,<br/>retry with backoff"]
APPROVAL -- "rejected, canceled,<br/>or timed out" --> CANCELED
CAPTURE -- "succeeds" --> SETTLE["settle: split the fee,<br/>post the ledger,<br/>notify captured"]
CAPTURE -- "retries exhausted" --> COMPENSATE["compensate: refund,<br/>post the correction,<br/>notify reversed"]
SETTLE --> DONE(["done"])
COMPENSATE --> DONE
Every arrow above is a point where a planted bug can change what actually happens: skip the approval gate on a large payment, retry capture one time too many, refund the wrong amount, or notify the wrong webhook kind. The mutation score is a measurement of how many of those changes the test suite would actually catch.
The words used throughout:
| Term | Meaning |
|---|---|
| mutant | a small deliberate bug planted in the code |
| killed / survived | killed: a test caught it (good). survived: no test noticed (bad) |
| no coverage | the bug never even ran, so no test had the chance to catch it, worse than survived |
| workflow | a long running program Temporal keeps alive across crashes |
| sandbox | the sealed environment Temporal runs a workflow in, walled off from anything outside it |
| history | Temporal's diary of every step a workflow took |
| replay | rerunning code against a saved diary to check it makes the same moves |
| arm | a place the agent can spend a probe: a code region crossed with a concern family |
| planted gap | a blind spot the gym manufactures by deleting one defense, ground truth is machine derived |
| trace | the append only log of an agent run, enough to replay it byte for byte, no keys |
The loop:
flowchart LR
B["belief:<br/>where are the tests blind?"] --> A["pick the most<br/>promising arm"]
A --> M["plant one small<br/>bug there"]
M --> V{"deterministic verdict<br/>in milliseconds"}
V -- "killed" --> D["tests are strong there:<br/>belief goes down"]
V -- "survived, new gap signature" --> G["gap confirmed:<br/>belief goes up"]
D --> B
G --> B
Every arrow appends an event to the trace. The loop stops on budget or when it stops learning. The belief starts from a pre registered prior capped at four pseudo counts, so four contrary observations overturn any prior. The reward is decided by replay, never by an LLM.
To measure whether the learning helps, you need to know where the real gaps are.
make gym manufactures them so the answer cannot be fudged:
flowchart LR
H["8 recorded histories"] -- "delete one" --> F["filtered corpus"]
MU["same mutant"] --> K["replayed vs full corpus:<br/>killed"]
MU --> S["replayed vs filtered corpus:<br/>survived"]
K --> P["planted gap,<br/>derived by machine"]
S --> P
Nobody labels a gap by hand. The gym runs 82 real operator mutants over the payment workflow, three deletion configs, 10 seeds, 60 trials, three policies: thompson (the learner), static prior only, and uniform (the floor).
- Hypothesis 5 said the learner would halve uniform's median trials to the first planted gap. Measured: 23 against 39. A real 1.7x improvement, short of the 2x bar. Refuted.
- Hypothesis 6 said the learner would find at least as many planted gap regions as the priors alone, on average. Our first sweep read it refuted, until a review found the metric counting planted mutants where its own definition said regions. Under the pre registered region unit it is supported: a tie on average, with a strict win on one config, and the report prints beside the verdict that the margin is not resolvable at ten seeds.
The decomposition is the finding. Where the deleted defense wounds a region the priors already flag, the priors win (0.90 against 0.80): the prior is already pointing at the wound, so exploring dilutes it. Where it wounds a region the priors do not tag, the compensation sequence, the learner wins (1.80 against 1.70). Priors win exactly where priors are calibrated, learning wins exactly where priors are blind, and the learner beats the uniform floor on time to first gap, 23 against 39. They cover different failures.
- The judge is outside the learner. The reward is deterministic survival, decided by replay. The LLM judge only ranks survivors for a human afterward. A dependency gate makes this structural: the policy, reward, and stopping code cannot import the LLM client, and the agent cannot import the seeded bug map. Contamination is a lint failure, not a promise.
- Every run is auditable.
make agentwrites an append only trace plus the raw LLM transcripts and mutates only a disposable worktree. The committed flagship run did four rounds, confirmed four gap signatures, and spent 7.8 cents of a two dollar cap.make agent-replayreproduces it offline, byte for byte. - Confirmed means adjudicated. A survivor flagged for human review is never counted as a confirmed gap until a human rules on it.
The standard mutation tester scored these workflow tests at 1.99 percent, not because the tests are bad, but because of a mechanism problem that has nothing to do with Temporal being unusual and everything to do with a trick every mutation tester relies on running into a wall that Temporal puts up on purpose.
Here is that trick, in plain terms, no Temporal knowledge assumed. A mutation tester does not build 151 separate copies of your code, one per bug. That would be slow to build and slow to run. Instead it writes all 151 small bugs into one copy of the code at once, each one guarded by a check that asks a single shared switch, which bug, if any, is active right now. Before running the tests for bug number 12, the tester reaches into the running program and flips that switch to 12. The guarded code reads the switch, takes the buggy branch, and the tests either notice or they do not.
That reach only works if the tester and the code it is testing share the same room. Workflow code does not. Temporal has to be able to replay a workflow's diary years later, on a different machine, after a crash, and get the identical sequence of steps every single time, so it runs workflow code inside a sealed environment, the sandbox, that cannot see a clock, cannot see randomness, and cannot see any variable set from outside it, on purpose: anything that could make two runs of the same workflow disagree is walled off. The mutation tester's switch is one more thing sitting outside that wall. It gets flipped in the ordinary test process. The workflow reads its own copy of the switch from inside the sandbox, and that copy was never touched. So the workflow keeps running the original, unmutated code, no matter which of the 151 bugs the tester believes it is currently checking.
That is what the diagram below shows, and it is why the report calls 143 of the 151 bugs no coverage rather than survived. Survived would mean a real bug ran and no test caught it. No coverage means the bug never got the chance to run at all, so there was nothing for a test to catch. The 1.99 percent score is not a verdict on the tests. It is a measurement of how rarely the switch actually reached the code.
flowchart LR
subgraph T["your test process"]
SW["mutant switch: ON"]
end
subgraph SB["Temporal's sealed sandbox"]
WF["the workflow reads its own globals:<br/>the switch never arrives"]
end
SW -. "blocked by the seal" .-> WF
143 of 151 planted bugs were never switched on, so the score measured almost nothing. This project patches the switch directly into the code bundle that enters the sandbox, and judges every mutant by replaying recorded histories, the same machinery Temporal itself uses to resume a crashed workflow. The sandbox strictness that broke the conventional tool is exactly what makes replay sound.
The economics of that trade are the real justification for replay, not just the coverage fix. Recording the 8 business scenarios, the happy path, a decline, an approval timeout, a retried capture, a compensated refund, and so on, is the one expensive step, done once, ever, against a real server. Judging a mutant against those recordings is the cheap step, done 151 times per run:
flowchart LR
subgraph Live["the alternative: run each mutant live"]
L1["spin up a worker<br/>and a temporal test server"] --> L2["run the whole payment flow<br/>for real: authorize, maybe wait<br/>for approval, capture, settle"]
L2 --> L3["the mutant switch usually never<br/>reaches the sandbox:<br/>no coverage, most of the time"]
end
subgraph Replay["what this project does: replay each mutant"]
R1["build the workflow bundle<br/>once, about 1.4 s"] --> R2["replay the 8 recorded scenarios,<br/>declined, timed out, retried,<br/>compensated, against it"]
R2 --> R3["about 118 ms per mutant,<br/>roughly 15 ms per scenario,<br/>no server, nothing to wait on"]
end
That is the performance difference in one picture: the live path pays for a server and a real wait on every mutant and, on top of that, usually cannot even switch the bug on. The replay path pays for one bundle, then a few recorded comparisons measured in milliseconds. 151 mutants replayed this way cost under 20 seconds of real computation. The same count run live is the 1.99 percent column two tables down, most of it spent starting infrastructure to test bugs that never actually ran.
The full causal story is docs/spikes/01-replay-instrument.md. The experiment
design and the two earlier refuted hypotheses are
docs/spikes/02-experiment-design-and-results.md.
| Measured on the same 151 workflow mutants | Conventional runner | Replay method |
|---|---|---|
| Planted bugs never even switched on | 143 | 0 |
| Workflow mutation score | 1.99 percent | 64.24 percent |
| Peak memory, full run (Linux container) | 2507 MB | 1732 MB |
| Wall clock, full run (a tie, and we say so) | 139 s | 150 s |
Every bug meets the examiner that can see it, and the layering below is machine enforced, which is what makes a mutant's file path a valid routing key:
| Layer | Contains | Judged by |
|---|---|---|
src/domain/ |
pure decisions: money math, states | property tests (generated input) |
src/workflows/ |
orchestration: order, timers, signals | replay against recordings |
src/activities/ |
side effects: charge, ledger, webhook | contract tests (mocked context) |
What replay cannot see is stated and routed, not hidden: values and durations,
paths the recordings never took, and query handlers are all blind spots by
design, measured in the report as leak through and edge adequacy, and covered by
the other tiers plus the AI adversary and judge. The blind spots are documented
in docs/spikes/01-replay-instrument.md. The AI layer, its measured bias
checks, and the agent in its gym are docs/spikes/03-ai-adversary-judge-and-agent.md.
The four habits you can adopt tomorrow without any of this open
docs/spikes/04-practical-notes.md.
src/
agent/ the budgeted experiment loop: arms, priors, reward, trace
gym/ the gap gym: deletion episodes, machine derived ground truth
workflows/ the target system's orchestration (the code being tested)
domain/ the target system's pure decision logic
activities/ the target system's side effect adapters
bugs/ seeded bug variants used by the demonstration tests only
runner/ the instrument: Stryker plugin, oracle router, activation, replay tier
oracle/ the instrument: corpus replay and drift checks
report/ the analysis: scores, honesty metrics, report rendering
semantic/ the analysis: the AI adversary and judge
tests/ one directory per tier, mirroring the routing table
corpus/ the recorded histories plus their manifest
runs/ recorded agent runs, the flagship run is committed for the audit
tools/ demo, gym, agent runs, corpus generation, drift gate, report
docs/ spike records: the instrument, the experiment, the AI layer, practical notes
site/ the generated experiment report page
- TypeScript with
pnpm, tested byvitestandfast-check. @stryker-mutator/coreplants the mutants. The replay runner is a custom Stryker test runner plugin. The Temporal SDK provides bundling and replay.dependency-cruiserenforces the layering and the learner isolation rules.- The AI layer uses the official Anthropic and OpenAI SDKs, capped by a spend budget.
MIT.