Proposal: finding flaky tests

[Zaba505]

Note

Revision 1 answers both open questions with (a): a flake with an open report is listed and nothing is posted, and a sweep is a fixed count.

It also makes two changes:

• Bug reports are posted by a Go command, flaky report. It renders the agent's description and notes into a fixed template with the test's results, checks them, and creates the discussion. The command in this post becomes flaky stress.
• Everything flaky runs is discovered each time it runs: the modules, packages, build tags, tests and walkthroughs. So every walkthrough's functional tests find their cluster from KIND_CLUSTER_NAME, and the billing walkthrough's run without test.sh.

All the open questions are answered.

Note

This isn't part of the humus re-evaluation. The charter, #629, migrates humus's old features, and this is a tool for the repository's maintainers. It has no wave, and the charter doesn't change.

Summary

flaky runs the repository's tests until they fail, so a flaky test is found before it turns CI red. A maintainer runs it by hand, on their own machine, which has many more cores than a GitHub runner.

• It runs every level.

  • The unit tests.
  • The testcontainers tests.
  • Each walkthrough's functional tests, against a kind cluster of its own.

• It varies what has found flakes before.

  • The CPU count: each run gets one -cpu value, and its process is pinned to that many cores.
  • The load: a run has its cores to itself, or shares them with another run.

  It runs every test the same number of times at each pair of the two, so the rates can be compared.

• It schedules the runs as a DAG, packed into lanes of cores.

  • It builds every binary and image first.
  • Then it sweeps every package.
  • Then it runs each test that failed or hung alone, many more times, to measure its rate.

  A lane never runs more than its load on a core, and no run waits forever for room.

• It reports each flaky test. For each one it gives:

  • its failures and hangs, at each CPU count and load;
  • its first failure;
  • the command that reproduces it.

• The find-flakes skill acts on the report. It runs flaky on the default branch, then looks for a Bugs discussion or story that already names each flaky test. For each one with none, it posts a Bugs discussion, and triage-bug takes it from there.

• It reports a flake. It doesn't hide one. It retries nothing and skips nothing.

$ flaky ./grpc/grpchealth ./example/wallet/...
built 9 test binaries and 1 image in 2m14s
swept 9 packages: 3,000 unit runs and 150 testcontainers runs of each, 10 functional runs of example/wallet
confirmed 1 test in 6,000 runs

grpc/grpchealth  TestX/a_case
              load 1        load 2
    cpu 1     0 / 1,000     0 / 1,000
    cpu 2     1 / 1,000     3 / 1,000
    cpu 4     2 / 1,000     5 / 1,000      11 failures, 0 hangs in 6,000 runs
    server_test.go:270: …
    output:    /tmp/flaky-20261021T1903/grpc/grpchealth/TestX/a_case/
    reproduce: flaky -run '^TestX$/^a_case$' -cpu 4 -load 2 -count 1000 ./grpc/grpchealth
               taskset -c 0-3 go -C grpc test -race -cover -count=1000 -cpu 4 -run '^TestX$/^a_case$' ./grpchealth/

every other test passed every run

The output is illustrative. TestX stands in for a real test.

Background

• Flakes are found once they turn CI red. A job fails on a change that didn't touch its module. Someone re-runs it, then reproduces the failure by hand with go test -c -race -cover, -count=3000 and -cpu 1,2,4, sometimes in several processes at once. That is how these were found:

  • the stories #866, #879, #927, #940 and #943;
  • the bugs #959, #1094, #1098, #1250, #1254 and #1255.

• They are common. In the 7 days to 2026-10-21, build.yaml ran 274 times on pushes to main.

  • 41 of those runs failed, each after the merge queue had passed the same change.
  • orderevents' TestInit_produces_the_order_s_events_keyed_by_its_ID_in_order alone failed 10 of them before it was reported as example/orderstatus: the orderevents tests read back no events when the reader's first Fetch leaves out the events' partition #1255.
  • 34 of 153 merge-queue runs failed.
  • 68 of 526 pull-request runs were re-run.

• The CPU count found some of them.

  • #866 was reproduced at -cpu 1,2,4.
  • #959's signal tests failed at -cpu=4: 21 failures in 48,000 iterations.

• Load found others.

  • #879's hangs were reproduced in 8 processes at once.
  • #1121's hang was reproduced at 12 runs at a time.
  • #1254 failed 10 times in 15 with another suite running alongside.

  -test.cpu sets GOMAXPROCS, but on a 32-core machine a run at -cpu 4 still has every core to itself. A GitHub runner has 4 vCPUs, shared by the test and its containers.

• A flake can be a real bug.

  • #959: a SIGTERM could be reported as a Service returning.
  • #1250: a Database's rotation could be lost for good.
  • #1254 was a bug in grpcserver, now #1258.
  • #879's hang was in franz-go (twmb/franz-go#1508).

  The others were races in the tests themselves. Only triage tells the two apart.

• golang.org/x/tools/cmd/stress runs one command in a loop, several processes at a time. It saves each failure's output, and kills a process that runs past a timeout. It doesn't:

  • know which test failed;
  • build the binaries or sweep the modules;
  • pin a run to cores;
  • vary the load.

Proposed API

The tool adds no API to humus. What it adds is a command, its module, and a skill.

The module

tools/flaky/
├── go.mod      github.com/z5labs/humus/tools/flaky
├── main.go     the flaky command
├── dag/        the tasks, what each waits on, and the barriers
├── lanes/      packing runs into lanes of cores
├── run/        running a test binary pinned to its cores, and reading its results
├── docker/     building images, pinning containers, and kind clusters
└── testdata/   packages whose tests fail or hang on cue

• The module isn't released. release-modules.sh skips tools/, as it skips example/.
• go-modules.sh lists it, so CI lints, builds and tests it with no change to build.yaml.
• It runs on Linux. It needs:
  • go and taskset;
  • for the testcontainers level, docker;
  • for the functional level, kind and kubectl, plus whatever up.sh needs.
• It warns when the local Go isn't the version build.yaml sets.

The command

flaky [-tags unit,testcontainers,functional] [-cpu 1,2,4] [-load 1,2]
      [-count n] [-confirm n] [-timeout d] [-keep n] [-containers n] [-clusters n]
      [-shuffle] [-run regexp] [-skip regexp] [-o dir] [-json] [packages]

• Packages are patterns from the repository's root, such as ./grpc/... or ./example/wallet/app. With none, it runs every module go-modules.sh lists.
• -tags picks the levels. The default is all three:
  • unit runs the untagged tests, as the Test jobs do;
  • testcontainers builds with that tag, as the testcontainers jobs do;
  • functional runs each walkthrough's functional/ tests against a kind cluster, as the kind jobs do.
• -cpu and -load are the two dimensions it stresses. -count and -confirm apply to each pair, so -count 500 -cpu 1,2,4 -load 1,2 is 3,000 runs of each package, 500 at each pair.
• With -run, it skips the sweep. It runs the matching tests -count times at each pair. That is the check a flaky test's story asks for.

The work

Each piece of work is a task, and a task waits on others:

build(pkg, tag) ──────┐
image(walkthrough) ───┴─▶ built ─┬─▶ sweep(pkg, cpu, load) ×count ───────────────────────────┐
                                 └─▶ up(walkthrough) ─▶ sweep(walkthrough) ×count ─▶ down ─────┴─▶ swept ─▶ confirm(test, cpu, load) ×confirm

• build builds one package's test binary with go test -c -race -cover and the level's tag, CI's flags.
• image builds one walkthrough's image with docker build, as the kind job does before up.sh.
• built is a barrier. No run starts until every build and image is done, since builds aren't pinned and would load the lanes' cores.
• A sweep run is one run of a package's test binary at one CPU count and load.
• up, down and confirm are described under "Clusters" and "Confirming" below.
• swept is a barrier after the last sweep run.

So the graph is two barriers and the clusters' lifecycles. What uses the cores well is how the runs are packed, under "Lanes" below.

Lanes

• The lanes. -keep cores (8 by default) are kept back for the tool, dockerd and the testcontainers containers. The rest are split into lanes as wide as the largest -cpu value. On 32 cores with -cpu 1,2,4, that is 6 lanes of 4.

• Each run is pinned to its cores. A run at -cpu c gets c cores in a lane and runs under taskset on exactly those cores, with -test.cpu c. So both GOMAXPROCS and runtime.NumCPU() are c, as on a runner with c vCPUs.

• A lane runs at one load at a time. At load L, each of the lane's cores has up to L runs pinned to it. At load 1, a lane of 4 holds one run at -cpu 4, two at -cpu 2, or four at -cpu 1. At load 2 it holds twice that.

  Load counts the runs pinned to a core, not their CPU use: a test that waits on a timer uses none.

• The queue.

  • The pair furthest behind its count goes first: one (package, CPU count, load). So every test has the same share of its runs done at any moment.
  • Ties go to the longest runs, measured as they finish.
  • Clusters start before everything else, since each holds its lane for the longest.

• Taking a task.

  • When a lane has room, it takes the queue's next task.
  • If the task doesn't fit, because it's wider than the lane's free cores or at another load, the lane holds it. It takes nothing else until enough of its runs have ended.
  • The other lanes keep taking the tasks after it.

  So a lane holds at most one task, and no task waits forever.

A run

• How it's invoked. In the package's directory, as go test runs it:

  taskset -c <cores> <pkg>.test -test.count=1 -test.cpu=<c> -test.timeout=<t> -test.v=test2json [-test.shuffle=<seed>] [-test.run=…]
  

  -test.v=test2json lets flaky read each test's result from the run's output. With -shuffle, each run shuffles with a seed of its own, kept with its result.

• A hang. At -timeout, the binary panics on its own, dumps every goroutine, and names the tests still running. Each of those tests counts a hang. A run still going 30s later is killed.

• Failing outside a test. A run that fails outside any test, in TestMain or a package's init, counts against the package.

Containers

• The testcontainers containers run on the kept-back cores. The containers that a testcontainers test starts, such as Kafka, PostgreSQL or k3s, are started by dockerd, so pinning the test's process doesn't pin them.
  • flaky watches Docker's events, and pins each container labelled org.testcontainers to the kept-back cores as it starts.
  • So the lanes stay clean. Only the code under test, which runs in the test's process, gets the run's CPU count and load.
• -containers caps how many testcontainers runs go at once. The default is 4.

Clusters

• The work.
  • up creates a kind cluster named flaky-<walkthrough>, and pins its node container to one lane's 4 cores with docker update --cpuset-cpus.
  • It runs the walkthrough's up.sh with KIND_CLUSTER_NAME set. up.sh deploys onto the cluster that's already there.
  • It waits for the Deployments to roll out, as the kind job does.
  • The functional runs follow, one at a time, with -context kind-flaky-<walkthrough>.
  • down deletes the cluster.
• A cluster holds its lane at load 1, from up to down. Its functional runs are pinned to the same 4 cores. That is CI's condition: on a 4-vCPU runner, the cluster and the test share the cores.
  • Its CPU count isn't varied, since below 4 cores the whole cluster starves: its control plane, its LGTM stack and its broker or database.
  • Neither is its load.
• A cluster serves one run at a time. A case like ARolloutOfTheLedger rolls the Deployments out, and another run on the same cluster would see it.
• -clusters caps how many clusters are up at once, for their memory. The default is 2. Clusters with different names can run side by side, since nothing maps a host port and the port-forwards use ephemeral ports.

Confirming

• Confirm runs wait for swept. A confirm task is added to the graph as soon as a sweep run reports a failure or a hang, but it doesn't start until every sweep run has ended.
• Each confirm runs one test alone, by its own -run pattern, -confirm times at every pair of CPU count and load. That includes the pairs where it never failed, since whether the rate depends on them is a clue for triage.
• A functional test gets a fresh cluster for its confirm runs.

Counts

These are the defaults for each pair of CPU count and load:

Level	-cpu	-load	-count	-confirm	-timeout
unit	1, 2, 4	1, 2	500	1,000	2m
testcontainers	1, 2, 4	1, 2	25	50	10m
functional	4	1	10	50	15m

The unit sweep keeps the 3,000 runs of each package that the stories have used.

The report

• It reports each test that failed or hung:
  • a table of its failures and hangs out of its runs, at each pair of CPU count and load;
  • its first failure's lines;
  • the directory its failures' whole output was saved to, under -o;
  • the command that reproduces it with flaky, at the pair where it failed most;
  • the same with go test under taskset, which can't reproduce a load.
• A test that failed every run at some pair is listed apart, as failing rather than flaky. On a default branch that CI passes, that is most likely the local machine's doing.
• Then it reports the rest: the number of tests that passed every run, and how long each phase took.
• -json writes the same report as JSON, for the skill.
• It exits 1 if any test failed or hung.

The find-flakes skill

.claude/skills/find-flakes/SKILL.md, run as /find-flakes [flaky's flags and packages] [--dry-run].

1. It runs flaky on the default branch. It makes a detached worktree of the default branch in the scratchpad, as triage-bug does, then builds flaky from it and runs it there with -json. So no local change is what it measures.
2. It sorts what flaky found:
   • a test that failed every run: it lists the test in its report and posts nothing;
   • a test with an open Bugs discussion or story that names it: it lists the test in its report and posts nothing;
   • a test whose last report is closed: it posts a Bugs discussion that links the closed one, since the test fails again at a commit after it;
   • a test nobody has reported: it posts a Bugs discussion.
3. Each Bugs discussion is written as example/databases: a Secret deleted while a reconcile is in flight comes back with the old password, so the rotation is lost #1250 and example/wallet: a rollout of the ledger can end a watch with LEDGER_UNAVAILABLE, when the wallet opens the ledger's watch again on the pod that's shutting down #1254 were.
   • It is titled <package>: <test> fails intermittently.
   • ## What happens gives:
     • the commit, the Go version and the machine's core count;
     • the table of failures and hangs at each pair of CPU count and load;
     • the first failure's lines;
     • the two commands that reproduce it.
   • ## Why says only that it hasn't been triaged.
   • ## Expected says the test passes on every run.
   • ## References links any earlier report.
4. It doesn't look for the cause. That is triage-bug's job. The skill ends by listing the discussions it posted, for /triage-bug.
5. --dry-run writes the discussions to the scratchpad rather than posting them.

Changes to other files

• .github/scripts/release-modules.sh skips every module under tools/, with a case in release-modules_test.sh.
• triage-bug: a story for a flaky test asks for flaky -run '<the test>' -count <n> <package> to report no failures or hangs at any pair of CPU count and load. Its pull request gives the output.

Design decisions

• A maintainer runs it, on their own machine. That machine has many more cores than a GitHub runner, so it can afford thousands of runs at each pair of CPU count and load. No workflow runs it.
• Each run is pinned to the cores it's given. -test.cpu alone sets GOMAXPROCS but leaves every core free, so it doesn't reproduce a runner with 4 vCPUs. taskset makes both GOMAXPROCS and runtime.NumCPU() match the CPU count. That is also why each run gets one -cpu value: -test.cpu 1,2,4 would run all three in one process, which can only be pinned once.
• Load is a dimension of its own. story(kafka): the tests that hold a produce request until they end pass on every run #879, story(humustest): a TLS connection on the Host closes as on a real socket, even if its write deadline is reset during close_notify #1121 and example/wallet: a rollout of the ledger can end a watch with LEDGER_UNAVAILABLE, when the wallet opens the ledger's watch again on the pod that's shutting down #1254 were reproduced with runs competing for cores. Load is set by how many runs are pinned to each core, so a test's rate at load 1 and at load 2 can be compared.
• Lanes, rather than packing over every core.
  • Runs of different widths fragment free cores: runs at -cpu 1 fill each gap as it opens, and a run at -cpu 4 never finds 4 free cores together.
  • In a lane, only the run the lane holds is waiting, so nothing starves.
  • A lane needs no estimate of how long a run takes, which a backfilling packer would.
  • Holding one load at a time keeps a load-1 run from sharing its cores with a load-2 run.
• Builds and images go first, behind a barrier. They aren't pinned, so overlapping them with runs would load the lanes' cores and skew the results.
• Confirming waits for the whole sweep. The sweep's runs stay spread evenly over every test until it ends. Then every test that failed gets its rate at every pair.
• The containers run on the kept-back cores. The code under test runs in the test's process, and that process is what gets the run's CPU count and load. The containers are its dependencies. Pinning them to the run's own cores would need a Docker proxy for each run, since testcontainers-go's session label hashes the parent's PID, and every run has flaky as its parent. At -cpu 1, Kafka or k3s would then starve as well.
• A cluster gets 4 cores and load 1. That is CI's condition. With fewer cores, or with load 2, timeouts from starvation would drown out the real flakes.
• It retries nothing and skips nothing. gotestsum's --rerun-fails, or a quarantine list, would have hidden humus: a SIGTERM can be reported as a Service returning, when the Service returns as soon as it is cancelled #959, example/databases: a Secret deleted while a reconcile is in flight comes back with the old password, so the rotation is lost #1250 and example/wallet: a rollout of the ledger can end a watch with LEDGER_UNAVAILABLE, when the wallet opens the ledger's watch again on the pod that's shutting down #1254.
• It runs its own loop rather than wrapping x/tools/cmd/stress. The loop is the small part. Building, pinning, scheduling and reading each test's result are the rest, and stress does none of them.
• The skill posts Bugs discussions, not stories. A flake can be a race in the test or a bug in the code, and only triage tells which. The story that triage files closes the discussion, as every bug story does.

Testing

• The scheduler is tested inside synctest, with the runs replaced by an implementation of flaky's own runner interface that each test controls. The tests check that:

  • no core ever has more runs pinned to it than its lane's load, and a lane at load L holds only runs at load L;
  • a lane holds at most one task, which starts as soon as enough of the lane's runs have ended;
  • with any mix of widths and loads, every task starts;
  • each pair of package, CPU count and load runs exactly -count times;
  • no run starts before built, and no confirm before swept;
  • a cluster serves one run at a time, on its own lane.

• A run is tested on small packages under testdata/:

  • one test passes every run;
  • one fails when runtime.NumCPU() is 2;
  • one blocks forever when GOMAXPROCS is 4;
  • one fails in TestMain.

  So the tests can check a run's pinning and results, and they see the same failures and hangs every time. A shuffled package checks that a failure's seed reproduces it.

• Docker and kind are tested behind the testcontainers tag, in a CI job of their own that isn't required. One test checks that a container labelled org.testcontainers lands on the kept-back cores. Another checks that a cluster's node container lands on its lane.

• The skill is run with --dry-run on the default branch, against one package from each level.

Migration from today

Nothing to migrate. Today flakes are reproduced and reported by hand, as #1250 and #1254 were, and they still can be.

Open questions

1. What happens to a flake that already has an open report?

   • (a) It is listed in the skill's report, and nothing is posted.
   • (b) The skill comments on the open discussion or story with the new rates.

   I'd recommend (a). The maintainer sees the numbers in the report, and an open report doesn't collect a comment every time the skill runs.

2. How big is a sweep?

   • (a) A fixed count for each pair at each level, as in the table above.
   • (b) A time budget, such as -for 8h, shared out across the packages by how long each one's runs take.

   I'd recommend (a). Two sweeps then give comparable results, and a story's "passes n runs" means the same thing in each.

Out of scope

• Reading CI's history for failures.
• Running it in CI, whether on a schedule, on pull requests or after a red push to main. It runs on a maintainer's machine.
• Other operating systems. Pinning needs Linux.
• Varying the containers' cores, or a cluster's CPU count or load.
• Retrying, skipping or quarantining a flaky test.
• Finding a flake's cause, or fixing it. That is triage-bug's job, and the story it files.
• Tests that aren't Go, such as .github/scripts/*_test.sh.
• Keeping history between runs.

References

• The flaky-test stories: #866, #879, #927, #940, #943, #1099, #1121
• The bugs that began as flakes: #959, #1094, #1098, #1250, #1254, #1255
• golang.org/x/tools/cmd/stress
• cmd/test2json, which reads a test binary's -test.v=test2json output
• testing's flags: -test.cpu, -test.timeout and -test.shuffle
• taskset(1) and docker update --cpuset-cpus

[Zaba505]

Revision 1: both questions answered, reports posted by flaky report, and everything discovered

Both open questions are answered with (a):

1. A flake that already has an open report is listed in the skill's report, and nothing is posted.
2. A sweep is a fixed count for each pair of CPU count and load at each level, as in the table.

This revision also makes two changes:

• Bug reports are posted by a Go command, flaky report, not by the skill.

  • The skill writes only the parts that need judgment, a description and any notes, to files.
  • flaky report checks them, renders them into a fixed template with the test's results, and creates the discussion.

  So every report has the same title, sections and tables, whatever the agent writes.

• Everything flaky runs is discovered each time it runs: the modules, packages, build tags, tests and walkthroughs. Adding or removing any of them changes nothing in flaky. To make that hold for every walkthrough, two of them change:

  • Every walkthrough's functional tests find their cluster from KIND_CLUSTER_NAME.
  • The billing walkthrough's functional tests run without test.sh.

This revision adds no questions. Each section below replaces, or adds to, the part of the same name in the post. Everything else is unchanged.

Summary

The bullet "The find-flakes skill acts on the report" becomes:

• The find-flakes skill acts on the report. It runs flaky stress on the default branch, then works through each flaky test:

  1. It reads the test and its failures, and writes a description.
  2. It calls flaky report with the description.
  3. flaky report renders it into the Bugs template with the test's results, and posts it, unless an open report already names the test.

  triage-bug takes it from there.

This bullet is added after it:

• It finds what to run each time it runs. Modules, packages, build tags, tests and walkthroughs are all discovered, so nothing in flaky changes when they're added or removed.

The example's command becomes flaky stress ./grpc/grpchealth ./example/wallet/..., and its flaky -run … becomes flaky stress -run ….

The module

The tree becomes:

tools/flaky/
├── go.mod      github.com/z5labs/humus/tools/flaky
├── main.go     the flaky stress and flaky report commands
├── discover/   the modules, packages, tags, tests and walkthroughs, found on each run
├── dag/        the tasks, what each waits on, and the barriers
├── lanes/      packing runs into lanes of cores
├── run/        running a test binary pinned to its cores, and reading its results
├── docker/     building images, pinning containers, and kind clusters
├── report/     the Bugs template, its checks, and creating the discussion
└── testdata/   packages whose tests fail or hang on cue, and a repository to discover

The command

This section is renamed flaky stress, and its usage starts flaky stress. Its Packages and -tags bullets become:

• Packages are patterns from the repository's root, such as ./grpc/... or ./example/wallet/app. With none, it runs everything "Discovery" finds.
• -tags picks the levels, from the ones "Discovery" finds. The default is all of them.

Discovery

This section is added, after flaky stress. flaky keeps no list of what it runs. On every run it finds:

• The modules: every module go-modules.sh lists, as CI's matrices do.
• The levels:
  • The untagged unit level, plus one level for each build tag in a //go:build line of a tracked _test.go file. Today that is unit, testcontainers, functional, conformance and loopback.
  • testcontainers and functional are the two levels whose needs flaky knows: the containers on the kept-back cores, and a cluster.
  • Any other tag runs as the unit tests do, with that tag and unit's counts. So conformance and loopback are swept too. No CI job runs -tags loopback today.
• The packages: go list with each level's tag, in each module, gives the packages that have test files.
• The tests:
  • Each test binary lists its own top-level tests with -test.list.
  • A level's tests are the ones its binary lists and the untagged binary doesn't. So a run at the testcontainers level runs only the tests behind that tag, by -test.run, and no unit test runs at two levels.
  • Subtests are found as they run, from test2json.
• The walkthroughs:
  • A walkthrough is a module with a Dockerfile and a deploy/kind/up.sh.
  • Its image is built from the repository's root and tagged with its directory's name, as CI's kind jobs do.
  • Its functional tests are its packages at the functional level.

A walkthrough that doesn't keep to "The walkthroughs' contract" fails its up and is reported by name. It isn't skipped.

The walkthroughs' contract

This section is added, after "Discovery". A walkthrough is found and run, with no change to flaky, when it keeps to this:

• Its Dockerfile builds from the repository's root.
• Its deploy/kind/up.sh deploys onto the cluster that KIND_CLUSTER_NAME names, kind if it's unset, and creates the cluster only if it isn't there. All six do this today.
• Its functional tests are behind the functional tag, and run with no flags. They find the cluster's context as kind- followed by KIND_CLUSTER_NAME, and they reach the services themselves.

Two changes bring every walkthrough to it:

• Five walkthroughs' functional tests take -context, defaulting to kind-kind. They take its default from KIND_CLUSTER_NAME instead. CI's kind jobs don't change, since they leave it unset.
• The billing walkthrough's functional tests are run by deploy/kind/test.sh. It port-forwards PostgreSQL, WireMock, Tempo, Loki, Mimir and Grafana, reads two Secrets, and passes ten flags.
  • The tests do that themselves instead, as the other five do. They find the context the same way as the other five.
  • test.sh is deleted.
  • The billing kind job runs go -C example/billing test -count=1 -race -tags functional ./functional/, as the other kind jobs do.

Clusters

The bullet "The functional runs follow, one at a time, with -context kind-flaky-<walkthrough>" becomes:

• The functional runs follow, one at a time, with KIND_CLUSTER_NAME set, as it was for up.sh.

flaky report

This section is added, after "The report".

flaky report -results <file> -package <path> -test <name> -description <file> [-notes <file>] [-dry-run]

• The agent writes two files:

  • -description, required: what the test checks and how it fails, in the agent's words;
  • -notes, optional: anything else it saw, such as which pairs fail or a pattern in the output.

  Both are Markdown with no headings. flaky report refuses a file that is empty or has a heading, so neither can add a section or change one.

• The tool writes everything else. It renders it from the results of flaky stress -json that -results names. That covers:

  • the title;
  • the commit, the Go version and the core count;
  • the table;
  • the first failure;
  • both commands;
  • a link to the test's function at the commit.

• The template:

  Title: <package>: <test> fails intermittently
  
  ## What happens
  
  <the description>
  
  At [<commit>](<link>), with <Go version> on <n> cores, `flaky stress` ran it <runs> times:
  
  |       | load 1    | load 2    |
  |-------|-----------|-----------|
  | cpu 1 | 0 / 1,000 | 0 / 1,000 |
  | cpu 2 | 1 / 1,000 | 3 / 1,000 |
  | cpu 4 | 2 / 1,000 | 5 / 1,000 |
  
  That is <f> failures and <h> hangs. Its first failure:
  
  ```text
  <the first failure's lines>
  ```
  
  It reproduces with:
  
  ```sh
  flaky stress -run '<pattern>' -cpu <c> -load <l> -count <n> <package>
  taskset -c <cores> go -C <module> test -race -cover -count=<n> -cpu <c> -run '<pattern>' <package>
  ```
  
  <the notes, if any>
  
  ## Why
  
  This hasn't been triaged.
  
  ## Expected
  
  `<test>` passes on every run, at every CPU count and load.
  
  ## References
  
  - [`<test>`](<link to its function at the commit>)
  - <the closed report that last named it, if any>

• It checks, before it calls GitHub. It refuses, and posts nothing, when:

  • the results don't have the test, or the test passed every run;
  • the test failed every run at some pair, since that is failing rather than flaky;
  • an open Bugs discussion, or an open story, already names the test. Then it prints that report's link and exits 2, so the skill can list it.

  A closed report that names the test is linked under References.

• It creates the discussion. It finds the Bugs category by name, creates the discussion with gh api graphql, and prints its URL. With -dry-run, it prints the title and body instead, and posts nothing.

The find-flakes skill

The steps become:

1. It runs flaky stress on the default branch. It makes a detached worktree of the default branch in the scratchpad, as triage-bug does. It builds flaky from that worktree and runs flaky stress -json there. So no local change is what it measures.

2. It writes a report for each test that failed or hung.

   1. It reads the test, its failures' output under -o, and its results.
   2. It writes the description, and any notes, to files in the scratchpad.
   3. It calls flaky report with them.

   It never calls GitHub itself.

3. It sorts its report by what flaky report did:

   • posted, with the discussion's link;
   • refused because an open report names the test, with that report's link;
   • refused because the test failed every run.

4. It doesn't look for the cause. That is triage-bug's job. The skill ends by listing the discussions it posted, for /triage-bug.

5. --dry-run passes -dry-run to flaky report, and saves what it prints to the scratchpad.

Changes to other files

The triage-bug bullet's flaky -run becomes flaky stress -run. These are added:

• The functional tests of the databases, mailer, orderstatus, shoppingcart and wallet walkthroughs default -context from KIND_CLUSTER_NAME.
• The billing walkthrough's functional tests port-forward the services and read the Secrets themselves, and find the context from KIND_CLUSTER_NAME. Then:
  • example/billing/deploy/kind/test.sh is deleted;
  • the billing kind job in build.yaml runs go test, as the other kind jobs do;
  • its pages say how the tests are run now.

Design decisions

These are added:

• A Go command writes the report, not the agent.
  • The title, the sections, the table and the commands are rendered from the results. So they're the same in every report, and can't be mistyped.
  • The agent writes only what needs judgment, in files the command checks.
  • The command calls GitHub itself, so no report skips the checks.
• Discovery, not lists. CI's matrices already come from go-modules.sh. flaky also finds the tags, the tests and the walkthroughs, so it never falls behind the repository. A tag it doesn't know runs as unit tests do, rather than being skipped: a test run without its setup fails loudly, and a test skipped goes unnoticed.
• A level runs only its own tests. A binary built with -tags testcontainers holds the unit tests too, as CI runs them. Running them there would spend the level's few runs on tests the unit level runs 3,000 times.
• The walkthroughs share one contract. If billing kept a script of its own, flaky would need to know how to run that script, and so would any walkthrough that followed it.

Testing

These are added:

• Discovery is tested on a repository under testdata/, with several modules, tagged tests and a walkthrough. The tests check that adding a module, a tag, a test or a walkthrough there changes what is found, with no change to flaky.
• flaky report is tested with golden files: given results, a description and notes, the rendered title and body match the golden ones.
  • Each check is tested on its own: an empty file, a heading, a test that passed every run, one that failed every run, and an open report that names the test.
  • The two calls to GitHub, the search and the creation, sit behind an interface of the tool's own, which the tests implement.

Open questions

Both are answered, with (a), by this revision.

Out of scope

This is added:

• Editing a report, or commenting on one, after it's posted. flaky report only creates one.

[Zaba505]

Closing: finding flaky tests has landed

Revision 1's stories are all done. tools/flaky is a module of its own, and find-flakes is the repository skill that runs it:

• story(flaky): sweep every package's unit tests, each run pinned to the cores of its CPU count #1260 swept every package's unit tests, each run pinned to the cores of its CPU count (story(flaky): sweep every package's unit tests, each run pinned to the cores of its CPU count #1273);
• story(flaky): stress each test at load 1 and load 2, with runs sharing their cores #1261 stressed each test at load 1 and load 2, with runs sharing their cores (story(flaky): stress each test at load 1 and load 2, with runs sharing their cores #1280);
• story(flaky): confirm each test that failed in the sweep, at every CPU count and load #1262 confirmed each test that failed in the sweep, at every CPU count and load (story(flaky): confirm each test that failed in the sweep, at every CPU count and load #1284);
• story(flaky): shuffle each run's tests with a seed of its own, kept with its result #1263 shuffled each run's tests with a seed of its own, kept with its result (story(flaky): shuffle each run's tests with a seed of its own, kept with its result #1286);
• story(flaky): stress the testcontainers tests, with their containers on the kept-back cores #1264 stressed the testcontainers tests, with their containers on the kept-back cores (story(flaky): stress the testcontainers tests, with their containers on the kept-back cores #1287);
• story(example): the walkthroughs' functional tests find their kind cluster from KIND_CLUSTER_NAME #1265 had the walkthroughs' functional tests find their kind cluster from KIND_CLUSTER_NAME (story(example): the walkthroughs' functional tests find their kind cluster from KIND_CLUSTER_NAME #1288), and story(example): the billing walkthrough's functional tests reach its services themselves, with no test.sh #1266 had billing's reach its services themselves, with no test.sh (story(example): the billing walkthrough's functional tests reach its services themselves, with no test.sh #1289);
• story(flaky): stress each walkthrough's functional tests against a kind cluster pinned to one lane #1267 stressed each walkthrough's functional tests against a kind cluster pinned to one lane (story(flaky): stress each walkthrough's functional tests against a kind cluster pinned to one lane #1290);
• story(flaky): stress every build tag the tests use, as a level of its own #1268 made every build tag the tests use a level of its own (story(flaky): stress every build tag the tests use, as a level of its own #1291);
• story(flaky): post a flaky test's Bugs discussion from a fixed template with flaky report #1269 posted a flaky test's Bugs discussion from a fixed template with flaky report (story(flaky): post a flaky test's Bugs discussion from a fixed template with flaky report #1292);
• story(ai): find flaky tests on the default branch and report each with a find-flakes skill #1270 added the find-flakes skill (story(ai): find flaky tests on the default branch and report each with a find-flakes skill #1294);
• story(ai): triage-bug's stories for flaky tests ask for flaky stress to pass at every CPU count and load #1271 had triage-bug's stories for flaky tests ask for flaky stress to pass at every CPU count and load (story(ai): triage-bug's stories for flaky tests ask for flaky stress to pass at every CPU count and load #1297).

The stories were checked by the repository's verify commands and unit tests. No story ran the tool end to end over the repository, so #1270's trial run is left to the first real run. Bugs found in that run will be filed as new reports.

The milestone "Finding flaky tests" is closed with this discussion.