CI test management plan

CI test management: plan

Issue #369. Status: done (2026-11-09). Steps 1 and 3 in #390, 2 in #391, 4 in #392 (its first flake fixed in #394), 5 in #395 and #397, 6 in #398; main requires CI and merges through the merge queue. Replayed with CI's own map, test selection runs 72% of the suite's test time over 278 past pull requests, and missed none of 18 failing runs.

What a pull request runs, and how CI keeps the suite trustworthy: run less on pull requests where history says it loses nothing, keep the full matrix before anything reaches main, and keep flaky tests out of the gate without losing what they check.

Where CI is today

Measured over the last 20 completed CI runs (2026-11-05 to 2026-11-08):

• 31 jobs, about 250 runner-minutes a run. Every pull request runs all of them, a one-line docs change included.
• Wall time 12–21 minutes when nothing queues, set by the longest share (about 15 minutes; shares are dealt by test-durations.json).
• Queueing is the real cost. With several pull requests open on 2026-11-05, jobs waited 27–33 minutes for a runner before starting: the runners are limited by concurrency, and every run asks for 31 of them.
• Only lint and docs are required checks. A pull request with a red test job can be merged; "merge when green" is a habit, not a rule GitHub enforces.

Where the runner-minutes go (median job times):

Jobs	Count	Runner-minutes a run
tests, Python 3.11/3.12/3.13 in four shares each	12	~130
ids-i64 (dataset tests with 64-bit ids), four shares	4	~41
metal, four shares on macOS	4	~41
l2-cells (L2 cell-fastest), two shares	2	~20
mpi, docs, warp, vtk-master, examples, wheels, packaging, lint	8	~15

In the last 100 pull request runs, 67 succeeded, 23 were cancelled by a newer push, 9 failed and 11 were re-runs of a commit.

The plan

Six steps, each useful alone, in this order.

1. The path filter, and one required check

• Land #368 (rebased): a changes job classifies the changed files (tools/ci_changes.py). Docs alone run lint and the docs build; an example, lint and the examples; anything in the packages, the build or the workflows, everything. A path no rule names counts as code. Pushes to main run everything. Replayed over 278 merged pull requests, 87% of today's test time.
• Add one aggregate job, CI, that needs every other job, runs if: always(), and fails if any of them failed or was cancelled (skipped is a pass). Make it the required check, in place of lint and docs. Skipped jobs then pass the gate, and a red test job blocks the merge.

2. One Python version in full, the other two quick

On pull requests, run the full suite on 3.13 only, with VTK and MFEM as today, and -m quick on 3.11 and 3.12 (one job each, without them). Pushes to main, and the merge queue once there is one (step 6), keep the full matrix. 3.13 is what Tack is developed with, and the version CI already gives VTK and MFEM. This saves about 70 of the 250 runner-minutes a run (eight shares of ~11 minutes become two quick jobs of ~7), and frees 6 runners a run.

What history says it would lose: of the 18 failing pull request runs from 2026-10-14 to 2026-11-05 (step 5's history), most failed on two or three Python versions; the three that failed on 3.13 alone are still caught, since 3.13 still runs in full. Five failed on 3.11 or 3.12 and not on 3.13:

Run	Failed on	File
vis/r4-gaps	3.12	test_viskores_datasets.py
perf/round-1	3.11, 3.12	test_viskores_datasets.py
cpu/contended-fan-out	3.11, 3.12	test_cpu_threading.py
core/multi-device	3.11, 3.12	test_multi_device_switching.py
cpu/scaling	3.11, 3.12	test_cpu_pool.py

The difference there was VTK more than the Python version: 3.13 is the one test job with VTK and MFEM, and a test can take another path without them. So the quick tier runs without them. test_cpu_threading.py and test_multi_device_switching.py are quick modules, and test_cpu_pool.py (0.6 s) becomes one; test_viskores_datasets.py (425 s) has quick tests but not all of them, and the history names files, not tests. At most the two test_viskores_datasets.py runs would have passed their pull request, and the push to main would have failed. Keep ids-i64 in full: in 3 of the 18 it was the only job that failed.

3. Timing tests in a job of their own

Tests that assert what the scheduler or the OS did (the timing marker and the other require_idle users, in test_cpu_threading.py and test_numa.py) run in their own job, not in the shares:

• the shares run -m "not timing"; the new job runs -m timing alone, on a runner with nothing else of Tack's on it;
• a failure is re-run once, and one that passes then is reported as a flake (an annotation on the run), not a failure;
• the require_idle users not yet marked timing get the marker.

They do catch real regressions (the CPU threading branches of October), so they stay in CI. They are also the suite's main source of flakes: test_timing_a_contended_kernel_settles_serial failed once on a docs-only pull request on 2026-11-08 (#385).

4. Flaky tests, tracked

A tool (tools/ci_failures.py, from the prototype's history/fetch.py) that reads recent CI runs and lists, per test:

• failures on pushes to main, where nothing should fail;
• failures that passed when the same commit was re-run (11 of the last 100 pull request runs were re-runs);
• failures on pull requests whose changed files cannot reach the test (step 5's map, once there is one).

Run it weekly (a scheduled workflow) or by hand; its report goes to an issue. A test found flaky gets fixed or marked; a quarantine marker that hides failures is not the aim.

5. Test selection by file, then by test

From the prototype's measurement (below): a per-test-file map of the repository files each test file uses, recorded on main, and each pull request running the test files its changes reach. By file first: 68% of today's test time, against 63% by test, with a simpler map and less exposure to the cached-result risk.

• The recorder as a test-only pytest plugin (sys.monitoring, plus hooks on Kernel.__call__, Kernel._lower and CallBindings.resolve, and a sitecustomize for subprocesses).
• The map refreshed on every push to main, by one Linux share, as an artifact the pull request runs download.
• The selection applied to the full-suite jobs (step 2's 3.13, ids-i64, l2-cells, Metal), and the shares re-dealt over the selected files by their weights. Without re-dealing, selection saves runner-minutes but not wall time.
• Rules for what the map cannot see (below).

6. A merge queue as the safety net

With GitHub's merge queue, pull requests run steps 1–5's reduced set and the queue runs the full matrix on the merge commit before it reaches main. The push to main already runs the full matrix after every merge today, so this moves that run in front of the merge rather than adding one: breakage is caught before main, not after. It matters once steps 2 and 5 make pull request runs partial. The queue runs on merge_group events, so the workflow needs that trigger, and step 1's CI check is what the queue waits for.

Measurement for step 5 (2026-11-05)

How it was measured

• The map: the whole CPU suite on main (dc4ca7a2), recorded per test. Each test's map lists:
  • the repository functions it ran, recorded with sys.monitoring (PY_START, each location disabled after its first hit and re-enabled at the next test);
  • the files of every kernel it launched, and of everything lowering inlined into those kernels. Python never runs kernel bodies, so these come from hooks on Kernel.__call__, Kernel._lower and CallBindings.resolve;
  • what its subprocesses ran, through a sitecustomize.
• Fixture setup is credited to every test in its file, since a module-scoped fixture is built during whichever test needs it first.
• The recording run: 6755 passed, 371 skipped, 6 min 17 s on 8 workers on an M1 Max. The overhead is negligible.
• The replay:
  • the 278 pull requests merged from 2026-10-14 to 2026-11-05, each diffed from its merge commit's first parent;
  • every CI run on a pull request in that period that failed with named tests: 18 runs, each diffed against main as it was when the run started.
• Weights: test-durations.json, with each file's seconds spread over its tests by their local durations.
• Rules:
  • a changed test file selects all of its tests;
  • the build and CI files (pyproject.toml, uv.lock, conftest.py, .github/, test-durations.json) run everything;
  • a file under packages/*/src/ that no test used runs everything;
  • a file the pull request adds selects nothing by itself; the changed files that use it select the tests.

Results

Average test time against today, 278 pull requests
Path filter alone (#368)	87%
Plus selection by test file	68%
Plus selection by test	63%

With selection by test, the 278 pull requests split as follows:

	Pull requests
No test job	37
Full run	81
Partial, under 20% of suite time	50
Partial, 20%–90%	32
Partial, 90% or more	78

Failures: none missed. 9 of the 18 failing runs got a partial selection, with 13 failing tests between them, and every one was selected. The vis failures would have been found by running 12–31% of the suite:

Run	Share of suite selected
polygon-fan-area	17%
merge-points-polyhedra	12%
vtk-polyhedral-split	12%
vtk/algorithms	12%
polylines failure, in polyhedra-page and project-cell-averages	31%

The core and runtime failures were selected at 99–100%. The other 9 runs would have run everything.

What keeps runs broad

• Files nearly every test really uses. In the 90%+ pull requests, the broadest changed file was:

  File	Pull requests	Share of suite that uses it
  dataset/views.py	16	96%
  runtime/cpu.py	11	99%
  dataset/topology.py	10	96%
  lang/array.py	9	99%

  The rest were dataset.py, shapes.py, _common.py, ghosts.py and units.py. Narrowing these would need function granularity: map a diff's changed lines to the functions they are in, and select the tests that ran those functions.

• Full runs came from ci.yml (28 pull requests, that period's CI work) and from tack/dataset/__init__.py (29). That file holds only module-level code, so no test is credited with it; usually the change is a new algorithm's export, whose implementing files are selected anyway. A rule for module-only files could recover most of those.

Risks

• A result cached in Python (an lru_cache, a module-level dataset) credits only the first test that computes it, so a later test that depends on it could be missed by selection. None of the 13 failures hit this, but the sample is small. Selection by file narrows it: a file's tests share one entry.
• A dependency a pull request adds is unknown to main's map. The merge queue's full run (step 6), or until then the push to main, catches it.
• Metal's map. The map comes from CPU runs, so files only Metal runs are unmapped and force a full run, which is safe but broad.
• Wall time is set mostly by the runner queue and the longest share, so it falls less than test time does; steps 1 and 2 cut the queue by asking for fewer runners.
• Step 2 rests on 18 failing runs. If a version-specific failure gets through to main, the push run (or the merge queue) catches it and the quick tier gains the tests that missed it.

Open questions

• Metal on pull requests: the full four shares on every code change, as now, or step 5's selection? Metal finds bugs that come from its having no f64, which the CPU does not (#382), so it stays on every code change until selection is trusted.
• Whether l2-cells (a layout variant of the dataset tests) can drop to the quick tier on pull requests, as the other Python versions do; it was never the only job to fail in the 18 runs.