A prototype harness for fault-injection tests in OpenShell

[mpereira]

Over the last few days I've been playing around with ideas for running OpenShell fault-injection tests. I came up with a prototype harness that runs declarative experiments. In an experiment you lay out a timeline. Some workloads run, and operations hit the system while they're running, like killing a process or pausing a container. Then you write down how you expect it to end. Which workloads finish? What did they write? What state is the system left in? The harness runs the whole thing and tells you whether you were right.

Every run also leaves the same kind of output behind: logs, metrics, a timeline of events and charts of all of it. A flow where three sandboxes, a gateway and an outage overlap is hard to follow in logs alone. On a chart you can see what happened when, and that makes it a lot easier to understand the problem.

It lives in e2e/chaos on my fork.

This post walks through it: we'll run the smallest experiment and look at what it left behind, and then run a more complex one and pick up the concepts as we go.

The smallest experiment

An experiment is a YAML file. Here's hello-world. One sandbox writes a file, and a check makes sure it did:

apiVersion: chaos/v1alpha1
id: hello-world
version: 1
title: A sandbox says hello
hypothesis: >
  One hello instance runs its script to exit code 0, and the file greeting in
  its output directory holds the line "hello, world".
description: >
  The smallest experiment: one workload with one instance, two checks and no
  faults. ...
default-adapter: openshell-docker
timeout: 5m

workloads:
  - name: hello
    description: Writes the line "hello, world" to the file greeting in its output directory.
    kind: shell
    instances: 1
    timeout: 2m
    run: |
      echo "hello, world" > "$CHAOS_EXPERIMENT_WORKLOAD_INSTANCE_OUTPUT_DIR/greeting"
    checks:
      - name: exited-ok
        description: "For each hello instance: the script exited with code 0."
        kind: exit-code
        equals: 0
      - name: said-hello
        description: 'For each hello instance: the file greeting holds the line "hello, world".'
        kind: shell
        run: |
          grep -qx "hello, world" "$CHAOS_EXPERIMENT_WORKLOAD_INSTANCE_OUTPUT_DIR/greeting"

flow: hello
...

There are already a few concepts in there.

A workload is something to run, and how many copies of it you want. Each copy is an instance, named <workload>-N. So hello runs as hello-0. The only kind of workload so far is shell. Its run script goes into each instance and runs there, not on the machine where the harness is. Every instance has an output directory the harness collects when it's done.

The harness doesn't care what an instance actually is. It just needs to be able to create one, ask how it's doing, collect its output and delete it. That could be:

• a process on a machine
• a Docker container
• a Kubernetes pod
• an OpenShell sandbox

The adapter decides which, and I'll get to adapters later. Every run in this post uses OpenShell sandboxes.

Checks are what you believe should be true afterwards. A workload check runs once per instance. exit-code looks at how the script ended. shell runs a script of yours, and exit 0 means the check passed.

Running it

You run an experiment with chaos run. It needs a gateway to talk to, and a mise task starts one for you:

mise run e2e:chaos e2e/chaos/experiments/hello-world.yaml

It builds the gateway, CLI and images, and starts a fresh Docker-backed gateway. Then it runs chaos run against it and tears the gateway down. When the run is done, it prints the check results and the verdict:

  ok    preflight/docker-reachable
  ok    preflight/pull-otel-collector-image
  ok    preflight/disk-space
  ok    preflight/images-present
  ok    preflight/cli-runs
  ok    preflight/jq-installed
  ok    preflight/gateway-up
  ok    hello-0/exited-ok
  ok    hello-0/said-hello
  ok    adapter/no-leaked-containers
  ok    adapter/nothing-paused
  ok    adapter/gateway-healthy
PASS
.../e2e/chaos/runs/hello-world/20261029T112646Z-2f79

chaos share turns a run into a Markdown page with charts. Here's the chart it drew for that run:

There isn't much in it, because there isn't much in the experiment. One sandbox starts, writes its greeting and exits 0. The first second goes to setup: the preflight checks run, and the adapter's observers start (more on those below). Then the flow starts the hello workload. Creating its sandbox with openshell sandbox create takes about a second, so hello-0 shows up at second 2. By the time the harness first asks how it's doing, the script is already done. (The axis labels repeat because the whole flow fits in two seconds.)

Observers record what the system does while the flow runs, apart from what the workloads write themselves. They start before the flow and stop after it. They come from the adapter, so every experiment on it gets them. The OpenShell Docker adapter has three:

• A metrics scraper. Every 2 seconds it reads the gateway's Prometheus /metrics endpoint and appends the scrape to system/metrics/gateway.jsonl. Each family's type goes in gateway.types.json. This run has 16 scrapes of 27 families. By the second one the gateway had handled 38 gRPC requests: CreateSandbox, six GetSandbox, ConnectSupervisor, DeleteSandbox and so on, each counted by method and status code.
• A watcher that runs docker events for this run's containers and saves them to system/watch/docker-events.jsonl.
• A watcher that runs an OpenTelemetry Collector. The gateway sends its traces there, and the collector's output goes into system/watch/otel.jsonl.

A watcher is just a command. The harness timestamps every line it prints and saves it, so anything you can tail can be a watcher.

Scraping every 2 seconds is a sample, not a recording. Take openshell_server_supervisor_sessions, the number of sandboxes with a connected supervisor. It reads 0 in all 16 scrapes of this run, because the sandbox came and went between two of them. In a longer experiment it follows the sandbox count.

The scraper keeps every family the gateway exports. An experiment can also list the few it's about under metrics:. The report draws those first, and the Grafana dashboard draws only those. metric checks can assert on them, like "the gateway's database readiness probe passed in the last scrape". You'll see one in the next experiment.

What came out

Start with the check lines above. Your two checks are in there: hello-0/exited-ok and hello-0/said-hello. So are a few you didn't write. Preflight checks run before anything else and make sure the machine can run an experiment at all. Adapter checks run after every experiment. They catch the stuff nobody thinks to check for: a leaked container, a paused process, an unhealthy gateway.

Above the check lines, the harness logs each event as it happens:

2026-10-29 12:26:46,122 run 20261029T112646Z-2f79 -> .../e2e/chaos/runs/hello-world/20261029T112646Z-2f79
2026-10-29 12:26:46,122 run.started {"experiment": "hello-world", "version": 1, "seed": 0, "adapter": "openshell-docker"}
2026-10-29 12:26:47,279 node.started {"node": "flow", "name": "hello"}
2026-10-29 12:26:48,359 instance.created {"workload": "hello", "instance": "hello-0", "system_id": "0c64c26a-1d99-4416-8b2d-c85a8120f04f"}
2026-10-29 12:26:48,412 instance.state {"workload": "hello", "instance": "hello-0", "state": "Succeeded", "reason": "MainProcessCompleted"}
2026-10-29 12:26:48,738 instance.collected {"workload": "hello", "instance": "hello-0", "exit_code": 0}
2026-10-29 12:26:48,952 instance.deleted {"workload": "hello", "instance": "hello-0"}
2026-10-29 12:26:48,952 node.finished {"node": "flow", "name": "hello"}
2026-10-29 12:27:19,295 check.started {"scope": "workload", "instance": "hello-0", "check": "exited-ok"}
2026-10-29 12:27:19,295 check.finished {"scope": "workload", "instance": "hello-0", "check": "exited-ok", "ok": true}
...
2026-10-29 12:27:19,455 run.finished {"verdict": "PASS"}

(I shortened the path and cut a few check events.)

Between node.finished and the first check, the adapter waits 30 seconds. That gives the gateway time to finish deleting containers before the adapter looks for leaked ones. That's why a run with two seconds of work takes 33.

Everything the run produced lands in one run directory:

e2e/chaos/runs/hello-world/20261029T112646Z-2f79/
./adapter.yaml
./experiment.yaml
./logs/collect.log
./logs/gateway.log
./logs/harness.log
./logs/watch-docker-events.log
./logs/watch-otel.log
./manifest.json
./report.html
./system/containers.json
./system/healthz.json
./system/metrics/gateway.jsonl
./system/metrics/gateway.types.json
./system/watch/docker-events.jsonl
./system/watch/otel.jsonl
./timeline.jsonl
./verdict.json
./workloads/hello/instances/hello-0/env
./workloads/hello/instances/hello-0/logs/sandbox.log
./workloads/hello/instances/hello-0/logs/supervisor.log
./workloads/hello/instances/hello-0/output/.chaos/attempts
./workloads/hello/instances/hello-0/output/.chaos/exit_code
./workloads/hello/instances/hello-0/output/greeting
./workloads/hello/run.sh
./workloads/hello/wrapper.sh

greeting holds hello, world. .chaos/exit_code holds 0. .chaos/attempts gets a line every time the script starts. Here there's one.

timeline.jsonl has the same events you saw scroll by, as JSON, with t in seconds since the start:

{"t": 0.001, "ts": "2026-10-29T11:26:46.122471Z", "event": "run.started", "experiment": "hello-world", "version": 1, "seed": 0, "adapter": "openshell-docker"}
...
{"t": 33.333, "ts": "2026-10-29T11:27:19.455033Z", "event": "run.finished", "verdict": "PASS"}

manifest.json pins what ran: the experiment's sha256, the OpenShell version, the git sha, Docker version and image digests. system/ holds what the observers recorded.

Great, a sandbox can write a file. Now let's break something.

A harder one: kill the gateway

There are ten experiments so far:

uv run --project e2e/chaos chaos list

gateway-graceful-restart (v2)
gateway-kill-keeps-state (v3)
gateway-sigterm-during-recovery (v1)
hello-world (v1)
policy-tighten-survives-gateway-restart (v3)
sandbox-container-restart (v2)
sandbox-stop-start (v2)
smoke (v1)
supervisor-kill (v2)
supervisor-pause (v2)

(list also prints each one's title and hypothesis. I cut those.)

Some of them reproduce known bugs. You can filter for those:

uv run --project e2e/chaos chaos list --issue NVIDIA/OpenShell#4910

gateway-kill-keeps-state (v3)
  Sandboxes survive a hard gateway kill
  issues: NVIDIA/OpenShell#4910
    After the gateway process gets SIGKILL, stays stopped for 15s and is started
    again, each counter instance's script runs to completion without being
    started again, and a sandbox created after every counter instance has ended
    runs its script to exit code 0.

policy-tighten-survives-gateway-restart (v3)
  A tightened policy stays in effect across a gateway restart
  issues: NVIDIA/OpenShell#4910
    ...

Let's run the first one. gateway-kill-keeps-state asks a simple question. If the gateway dies hard, do the sandboxes it was managing carry on?

uv run --project e2e/chaos chaos show gateway-kill-keeps-state

gateway-kill-keeps-state (v3)
Sandboxes survive a hard gateway kill
default-adapter: openshell-docker   timeout: 15m

hypothesis:
  After the gateway process gets SIGKILL, stays stopped for 15s and is started
  again, each counter instance's script runs to completion without being started
  again, and a sandbox created after every counter instance has ended runs its
  script to exit code 0.
...
workloads:
  counter x3 (timeout 5m)
    Counts from 1 to $TICKS, one tick every $INTERVAL seconds. Each tick appends
    the line "<tick> <unix time>" to the instance's journal, then saves the tick
    number atomically. If the script is started again in the same output
    directory, it continues from the saved tick plus one.
    check exited-ok:
      ...
    check journal-complete:
      For each counter instance: the journal has the ticks 1 to $TICKS, each
      once and in order, so no tick was skipped or repeated.
    check started-once:
      For each counter instance: .chaos/attempts has exactly 1 line, so the
      script ran once and was not started again.

  fresh-sandbox x1 (timeout 1m)
    ...

operations:
  gateway-down: outage
    on target gateway, select one for 15s
    Sends SIGKILL to the gateway process and keeps it stopped for 15s. Then runs
    the adapter's start hook, which waits up to 120s until the gateway's
    /healthz returns 200, and records operation.reverted. The gateway is
    unavailable for longer than 15s.

checks:
  gateway-recovered:
    The timeline has an operation.reverted event for gateway-down. The harness
    records it only after the restarted gateway answered /healthz with 200.

  database-healthy-after-restart:
    The gateway's database readiness probe passed in the last scrape of the run,
    after the restart.

expectations:
  NVIDIA/OpenShell#4910, known to fail:
    counter/exited-ok
    counter/journal-complete

Three counters tick from 1 to 120, once a second. If something restarts a counter, it picks up from its last saved tick. A journal with ticks 1 to 120, each once, means nothing was lost or doubled. One attempt means nothing restarted it behind our back.

Here's the part of the YAML that breaks things:

operations:
  - name: gateway-down
    uses: outage
    target: gateway
    select: one
    duration: 15s

flow:
  serial:
    - parallel:
        - counter
        - serial: [{wait: {workload: counter, condition: started}}, {delay: 20s}, gateway-down]
    - fresh-sandbox

An operation is something the experiment does to the system during the flow. You declare it in the experiment. You give it a name and say what it uses and what it acts on: its target. select says which of the target's candidates to hit, and some operations take a duration. Some operations are faults, which break something on purpose. Others just change the system's state, the way an operator would. To the harness they're the same thing.

Operations come from two places.

The harness has five built-in faults: restart, outage, kill, restart-in-place and freeze. The name says what happens, and the target decides how. A process gets a signal. A Docker container gets docker kill or docker pause. A Kubernetes workload loses its pods.

The adapter can add its own. Each one is a shell hook that makes the change, and usually a second hook that undoes it at the end. The OpenShell adapters have two:

• openshell-policy-set changes a running sandbox's policy.
• openshell-sandbox-restart stops a sandbox and starts it again.

Need one that isn't there? Add it to the adapter. It doesn't have to be a fault.

Unlike a workload's script, an operation doesn't run inside the instances. It runs where the harness runs and reaches into the system from outside. For a process, the harness sends the signal itself. For containers and pods, it runs docker or kubectl. An adapter operation is a hook the harness runs on that same machine.

Here, gateway-down uses outage on the gateway target. In the Docker adapter that target is the gateway process on the host. The harness sends it SIGKILL and keeps it down for 15 seconds. Then the target's start hook starts the gateway again and waits until /healthz returns 200. So the gateway is gone for more than 15 seconds. In this run it was about 18. While it's down the sandboxes keep running, but neither the CLI nor the harness can reach the gateway.

The flow says when things happen. It's a tree of serial, parallel, wait, delay and repeat. This one says: start the counters. Alongside them, wait until they're running, wait 20 more seconds and take the gateway down. When all that's done, create one fresh sandbox to see whether the restarted gateway can still make new ones.

A wait doesn't have to wait for a workload. It can also wait for a signal, which is a line showing up in a watcher's output. The adapter names the signal and gives the pattern to look for. The wait ends the next time a line matches, or a set delay after that. The OpenShell Docker adapter has one, reconcile-sweep. It matches the trace span the gateway sends at the end of each reconcile sweep. sandbox-stop-start uses it to aim at a race. A sweep starts 60 seconds after the previous one ended. So the flow waits for a sweep and restarts all the sandboxes 57.5 seconds later, just before the next one. It does that for 20 minutes:

- repeat:
    for: 20m
    serial:
      - wait: {signal: reconcile-sweep, delay: 57500ms, timeout: 3m}
      - restart

That turns a race you'd hit once in a blue moon into one the experiment is likely to hit several times per run. A watcher is any command, so a signal can match anything you can turn into lines. That could be a log file you tail, a container's logs, or a loop that prints when a metric crosses a threshold. There's no built-in condition on metrics, though.

Back to gateway-kill-keeps-state. I ran it the same way as hello-world:

mise run e2e:chaos e2e/chaos/experiments/gateway-kill-keeps-state.yaml

Here are the check results:

  ok    preflight/docker-reachable
  ...
  ok    preflight/gateway-up
  xfail counter-0/exited-ok (known: NVIDIA/OpenShell#4910)
          no exit code recorded
  xfail counter-0/journal-complete (known: NVIDIA/OpenShell#4910)
          < 39
          < 40
          < 41
          < 42
          want the ticks 1 to 120, each once, in order
  ok    counter-0/started-once
  ...
  ok    fresh-sandbox-0/exited-ok
  ok    experiment/gateway-recovered
  ok    adapter/no-leaked-containers
  ok    adapter/nothing-paused
  ok    adapter/gateway-healthy
XFAIL

(I copied these from chaos evaluate, which prints the same block as chaos run. I cut counter-1 and counter-2, which look exactly like counter-0.)

All three journals stop at 38:

36 1793275760
37 1793275761
38 1793275762

Tick 38 is 12:09:22 UTC. The timeline has the gateway coming back one second later:

{"t": 22.861, "ts": "2026-10-29T12:09:05.176973Z", "event": "operation.started", "operation": "gateway-down", "uses": "outage", "run": 1, "target": "gateway", "scope": "target", "instances": ["gateway"], "ids": ["65379"]}
{"t": 41.037, "ts": "2026-10-29T12:09:23.353772Z", "event": "target.restarted", "target": "gateway", "operation": "gateway-down", "pid": 66029}
{"t": 41.038, "ts": "2026-10-29T12:09:23.354671Z", "event": "operation.reverted", "operation": "gateway-down", "uses": "outage"}

So the sandboxes were fine while the gateway was dead. They kept counting the whole time. It's the gateway coming back that killed them. gateway.log says why:

Error:   × attachment denied: boundary process leaf: sandbox runtime is already bound
  │ to another supervisor process

The restarted gateway starts a new supervisor for each live sandbox. The sandbox runtime rejects it, because the old one is still attached. Then the driver stops the container. That's #4910.

The run took 1m 50s. Here's what it looked like, from chaos share:

The red band in the flow section is the outage. Each counter goes Unknown while the harness can't ask the gateway for its state. It goes back to Running when the gateway returns, then Pending, then Failed.

report.html has the same story plus the metrics:

And if you want to dig around, mise run e2e:chaos:grafana <run dir> loads one or more runs into a local Prometheus and Grafana:

Expectations and verdicts

Why XFAIL and not FAIL? Because of this bit:

expectations:
  - issue: NVIDIA/OpenShell#4910
    # The restarted gateway replaces each live supervisor, the sandbox runtime
    # rejects the replacement, and the driver stops the sandbox container.
    checks: [counter/exited-ok, counter/journal-complete]

An expectation ties checks to a known issue. The checks still say what should happen. The expectation says "and right now it doesn't, see this issue". That way the experiment doesn't have to lie about the behavior it wants just to stay green.

There are four verdicts:

• PASS (exit 0): every check passed.
• XFAIL (exit 0): the only failures are known ones. The known bugs are still there and nothing else broke.
• FAIL (exit 1): something failed that no expectation covers. Or a check that's known to fail passed. That's an XPASS: maybe someone fixed the bug, so go update the expectation.
• ERROR (exit 2): the harness, preflight or a hook broke. This says nothing about OpenShell.

Some bugs are races and don't show up every time. Mark an expectation kind: flaky and the check may fail or pass. Both are fine. sandbox-stop-start uses that for #4981, which triggers in about 1 of 72 starts.

Adapters

Apart from the default-adapter line, nothing in the experiments above mentions Docker. That's the point of the adapter. The harness itself knows nothing about OpenShell, or about sandboxes at all. An adapter binds it to a system that runs workloads. It's a YAML file of hooks. They say how to create an instance, ask its state, collect it and delete it. They also name the fault targets, what to scrape and what to check afterwards. The built-in faults work on processes, containers and Kubernetes pods. So supporting some other system means writing the same kind of file with different hooks.

There are two so far. Both are for OpenShell, so an instance is a sandbox:

• openshell-docker runs under e2e/with-docker-gateway.sh. Its targets are the gateway process and the sandbox and supervisor containers. It has the observers and the preflight and adapter checks you saw above. All the runs in this post used it.
• openshell-kubernetes targets pods. It loads and validates, but I haven't run it against a cluster yet, so no run directory from it exists.

Preflight and provisioning

Before an experiment touches anything, the harness runs preflight checks. They make sure the system is there and in a state worth testing. A failed preflight check skips the flow and ends the run with ERROR. That way a broken setup never looks like a failed experiment. You saw the Docker adapter's preflight checks at the start of both runs: Docker answers, the images are there, the CLI runs and the gateway is up. The Kubernetes adapter's checks make sure the API server answers and the gateway has a ready replica. An experiment can add its own, which run after the adapter's. They're shell scripts too.

Preflight also tells you when an experiment can't run on an adapter at all. sandbox-container-restart restarts a sandbox's container in place. OpenShell's Kubernetes driver creates sandbox pods with restartPolicy: Never, which tells Kubernetes to never restart their containers. So on Kubernetes that fault can't happen. On openshell-kubernetes, chaos preflight and chaos run report the experiment as n/a and exit 0 without running anything.

Preflight checks that the system exists, but nothing in the harness creates it yet. Today the wrappers do. e2e/with-docker-gateway.sh builds and starts the gateway, and e2e/with-kube-gateway.sh installs the chart into a cluster. The plan is to give adapters a provision and a deprovision hook. Say you want to run experiments on a real Kubernetes cluster in the cloud. chaos provision runs the adapter's hook, which uses Terraform to create the cluster and installs the gateway on it. If the adapter also manages the machines, Ansible sets them up. Then chaos run starts with preflight, which makes sure the cluster is in good shape, and the experiment runs. The experiment itself only says what it needs, and preflight checks it. chaos run never provisions anything on its own.

Mostly shell, on purpose

If you've been reading the YAML closely, you've seen shell a lot. Almost everything you write is a shell script. A workload's run is a script that runs in each instance. A shell check is a script that runs on the host against the run directory. The adapter is shell too. Its hooks create, poll, collect and delete instances, and start targets back up. Its operations and watchers are scripts as well. The harness runs them, enforces their timeouts and records what happened. It doesn't need to know what they're for.

That's deliberate. I didn't want to invent abstractions before I knew which ones I needed. A script can do anything, so a new kind of experiment doesn't have to wait for a new feature. When the same script keeps showing up, it can become its own kind. The few things that aren't shell today are the ones that were worth a name: exit-code and metric checks, and the built-in faults, which have to mean the same thing on Docker and Kubernetes.

The run directory is the source of truth

Checks only ever read the run directory. They never talk to the live system. So you can evaluate a run again later, offline, with no gateway around.

uv run --project e2e/chaos chaos evaluate e2e/chaos/runs/gateway-kill-keeps-state/20261029T120842Z-3b2c

That writes a new file under evaluations/ and never touches the original verdict.json.

You can also evaluate an old run with a newer experiment file. I added the database-healthy-after-restart metric check after this run, and the old run picked it up:

uv run --project e2e/chaos chaos evaluate --experiment e2e/chaos/experiments/gateway-kill-keeps-state.yaml e2e/chaos/runs/gateway-kill-keeps-state/20261029T120842Z-3b2c

  evaluated with e2e/chaos/experiments/gateway-kill-keeps-state.yaml (sha256 e6e2e85c1097) and adapter openshell-docker (...)
  ...
  ok    experiment/database-healthy-after-restart
  ...
XFAIL

The same goes for the other commands. report, share, export, dashboard and the Grafana task all read the run directory. chaos --help lists them. Hand someone a run directory and they have everything you had.

Try it

The code is in a PR on my fork, on the branch feat/chaos-experiment-harness/mpereira. To check that your setup works, run:

mise run e2e:chaos:check

It runs hello-world and smoke, and fails unless both PASS. From there, pick any experiment:

mise run e2e:chaos e2e/chaos/experiments/gateway-kill-keeps-state.yaml

Right now the experiments reproduce these issues:

• Gateway restart leaves every sandbox in Error (StartFailed); --restart-policy always not honored #4910: a restarted gateway kills live sandboxes (gateway-kill-keeps-state, policy-tighten-survives-gateway-restart)
• bug(docker): sandbox stays in Error after its container restarts, and stop/start cannot recover it #4828: a sandbox whose container restarts goes to Error and never comes back (sandbox-container-restart)
• bug: reconciliation can fail a Docker sandbox during restart #4981, flaky: sandbox start sometimes fails because the reconcile sweep races it (sandbox-stop-start)
• Graceful gateway shutdown can lose a pending driver operation, leaving sandboxes undeletable (caller gone, or SIGTERM during startup recovery) #4409, flaky: a SIGTERM during the gateway's startup recovery doesn't stop it cleanly (gateway-sigterm-during-recovery)

Feedback welcome

This is just a prototype I built because I was interested in doing fault testing on OpenShell. I'd love to hear what you think of the ideas. Maybe some of them are useful and some aren't.

The important part is the experiment model, not how the prototype implements it. By the model I mean the workloads, operations, flow, checks and expectations, plus the adapter that binds them to a system. The prototype is Python and shell because that was the quickest way to try the idea. A real implementation could just as well sit on top of OpenShell's Rust testkit in tests/testkit.

Here's where I think it could help:

• CI. The experiments can run as end-to-end tests. When someone fixes a known issue, its experiment's XFAIL turns into an XPASS. That fails the run until someone updates the expectation.
• Reproducing reported issues. A bug report can become an experiment that reproduces it, on one machine or on Kubernetes. That includes the flaky ones that only show up after a particular sequence of events, like the reconcile race in bug: reconciliation can fail a Docker sandbox during restart #4981.
• Verification. Agents can use it too. An agent working on a fix can write an experiment that reproduces the bug. Then it changes the code and reruns the experiment until the checks pass. That shows the fix holds up under the same flow, timing and faults that broke it. The run directories from before and after are good evidence to put in the PR.
• Better bug reports. Whoever reports an issue can run the experiment themselves and zip up the run directory. Attached to the issue or PR, it carries all the logs and metrics from their own machine or infrastructure.