Proposal: a regression benchmark suite for comparing commits (results, speed, memory)

[pzeiger]

Hi all,

I would like to build a benchmark suite for abTEM and want to agree on what it is for before writing code. Short version: one command that runs a fixed set of standard simulations against two commits and reports, per case, whether the results changed, how the speed changed, and how the peak memory changed. Feedback on the goals and the case list is what I am after; the design details are collapsed at the end.

Why now

Every recent performance PR has shipped its own script and its own results table in its own format: #269, #289, #309, #346, #347, #366, and benchmarks/ on dev now holds 13 one-off scripts that each re-implement subprocess isolation, RSS/VRAM sampling, JSON output and a compare printer.

Result changes between versions are currently found by users, not by us. In #261 a version-to-version difference in diffraction intensities turned out to be a convention change plus a slice-assignment bug introduced by a potential-building speedup, found months later and fixed in #262. In #281 a user asked why 1.0.9 timed differently from 1.0.8, and we had no reference to answer with.

We have no way to say "this commit is bit-identical to the previous one on these 17 workloads and 8 % faster on GPU" except by trusting whatever script the PR author wrote that week.

What it should achieve

1. Make the question "did this change alter results, speed or memory?" answerable for any two commits with one command and one report, on CPU and GPU, in a format that is the same for every PR. Reviews of performance PRs then happen on shared evidence.
2. Catch unintended result changes at PR time instead of in user reports. "Result change" means: the same case produces arrays that differ beyond a stated tolerance, or whose shape, sampling or axes changed. A flagged change is not automatically a bug (the Diffraction code output being inconsistent depending on version of abTEM used? #261 convention fix was an improvement); the point is that it becomes visible and is accepted deliberately, which also gives us the changelog entry for free.
3. Check that equivalent code paths agree with each other within one commit: CPU vs GPU, lazy vs eager, PRISM vs multislice, different batch and chunk sizes. Those consistency gaps are tracked across commits too.

What it is not

• Not a validation of the physics against other codes or experiment. That is a different project with different reference data; the existing abTEM/abtem-benchmarks repository (scaling studies from 2022/23) or the docs are better homes for it.
• Not a micro-benchmark of individual kernels. Cases are whole user-facing workflows (a HAADF scan, a CBED pattern, a core-loss image), because that is what regresses in practice and what users notice.
• Not a public leaderboard or a claim about absolute performance. Numbers are only compared between two commits on the same machine.

What using it would look like

python -m abtem_bench run --ref dev --ref my-branch --device cpu,gpu --tier quick
python -m abtem_bench compare results/dev results/my-branch --md report.md

The report is a table with one row per case, device and tier, giving a result verdict (identical, within tolerance, drift, shape change, unsupported on one ref, error, OOM, timeout), the time ratio flagged against a measured noise floor, and the peak RSS/VRAM ratio. A self-check mode runs the same commit twice to measure that noise floor and to verify that CPU results are bit-reproducible under deterministic settings.

Proposed initial cases

Potential building (infinite and finite projection, CrystalPotential), HRTEM exit wave and image, SAED and CBED, HAADF scan, multi-detector scan (BF + ADF + segmented), 4D-STEM, flexible annular detector, thickness series, PRISM scan, frozen-phonon HAADF (fixed seed), core-loss image and PRISM core-loss (GPAW), Bloch-wave diffraction. Three sizes each: quick (CI, seconds), standard (minutes), large (GPU stress at the sizes the existing scripts use). Energy ensembles, tilt series, plasmons and magnetism would follow once the matrix is stable.

Questions where I want your input

1. Do you agree with the goals and non-goals above? In particular, should a result drift beyond tolerance block a merge (unless explicitly accepted), or only be reported?
2. Which workloads must be in the matrix? If you have a hot path you care about (C-PRISM, BiP-PRISM, PRISM-EELS, plasmon scattering, magnetic multislice), a case is about 20 lines and I would rather have it contributed than guessed.
3. Tolerances: is "float32 within rtol 1e-5 of the previous commit, float64 within 1e-10, frozen phonons looser with a fixed seed" the right default policy?
4. Where should it live? I propose benchmarks/ in the main repo, since cases track the API of the code they test and CI needs them in-tree, with the one-off scripts consolidated into it and then removed. The dormant abtem-benchmarks repo would stay as the home for scaling studies.
5. CI: I propose a label-triggered accuracy check (quick tier, CPU, PR head vs merge-base) on hosted runners, and speed/memory runs only on our own hardware. Should the accuracy check become a required check later?
6. Build vs adopt: I looked at asv and pytest-benchmark and propose a small custom harness with no new dependencies (details below). If someone has a strong preference for asv, now is the time.

Unless there are objections, I will start with a skeleton and three cases in about a week, so that the discussion can continue on a concrete PR.

Design details

Layers. Declarative cases (pure abtem API calls), a runner that executes each case in its own subprocess with time and memory meters, and a compare tool that works on a commit-independent result format (NumPy .npz plus JSON sidecars: arrays, dtype, shape, axes, invariants). The harness and the cases always come from the invoking checkout; only the abtem package is swapped per ref, so both refs run identical case code, and a case whose API is missing on one ref records "unsupported" instead of failing the run.

Swapping refs. abtem is pure Python, so each ref is a git worktree put on PYTHONPATH in a shared environment; an isolated uv venv per ref is the fallback for refs across the numpy 2 and zarr 3.1 pin changes. Refs are interleaved per case (A, B, A, B) so thermal drift hits both alike.

Determinism. Cases pass explicit gpts (never sampling, because grid.round-to-fast-fft can change derived grids between commits), pin max_batch and chunk sizes (the auto values depend on free memory), fix seeds, and the accuracy preset uses FFTW_ESTIMATE and one worker thread so CPU results are bit-reproducible. Auto-sized variants are reported but never flagged.

Meters. Wall time with GPU synchronisation, cold-start time (JIT, FFTW planning, CuPy kernel compilation) recorded separately from warm repeats, peak RSS by sampling plus ru_maxrss, peak VRAM from both the CuPy pool and the driver (to capture cuFFT workspace outside the pool).

Why not asv. It has no array-result comparison (only scalar tracking), no GPU memory, its per-commit environment management fights uv and CuPy builds, and --python=same cannot compare two commits. pytest-benchmark is in-process timing only. Both would still need the accuracy layer written by hand, so we would end up with two systems.

Milestones. (1) skeleton, three cases, self-check validated; (2) full case matrix and consistency pairs; (3) isolated environments, timeouts and OOM classification, report polish; (4) remove the legacy scripts, add the CI check, docs page.

Best,
Paul

🤖 Drafted with Claude Code, reviewed and edited by Paul.

[TomaSusi]

Very good suggestion, we definitely need this before we release all the current work in v1.1 – the exact Fresnel propagator change alone will break numerics (for the very good reason that it is simply more correct without real loss in performance), and it would be great to benchmark and be transparent about this.

I think I need to get back home from my conference marathon and rest a bit before I can think on this level, I don't want to leave this up to Claude :) But I should be able to provide feedback and allow this to be started next week. Thank you!

[TomaSusi]

Paul, now that I am back and have had time to think about this properly: yes, let's do it, and thank you for framing it as a discussion before writing the code.

The motivation is not hypothetical, and it is worth spelling out how bad the current situation actually is. The benchmarks/ directory is now about 4,500 lines across 13 scripts, and two of them — compare_benchmarks.py at 464 lines and the --compare mode inside benchmark_comparison.py at 368 lines — are independent implementations of the same comparison table, written weeks apart. Consolidating that would be worth doing even if we got nothing else out of it.

The layering you propose is the right one, and I want to point out the two details that I think will decide whether this succeeds, because you have already got both of them right. The first is that the harness and the case definitions come from the invoking checkout while only the abtem package is swapped, so both refs really do run identical case code. The second is that a case whose API does not exist on one ref records "unsupported" instead of bringing down the run. Suites like this usually die because one of those two things was not thought through at the start.

Before I answer the questions, there are two larger points I would like you to weigh, one about ordering and one about numerical precision.

Ordering: what v1.1 needs first is a comparison against v1.0.10
The two failures you cite, #261 and #281, were both about released versions, and both were found by users months after the fact. A live A/B between two checkouts cannot answer either of them retroactively, because it requires both refs to run in the same environment. With numpy>=2, zarr>=3.1 and the dask exclusions we now carry in pyproject.toml, standing v1.0.8 up next to dev is already an afternoon's work, and that only gets worse with every release.

What we concretely need from this suite in the next few months is a clear answer to: how does v1.1 differ numerically from v1.0.10, workload by workload, and which of those differences did we intend? The exact Fresnel propagator is the obvious example. It is now the default in abtem/multislice.py, it is simply more correct, it costs us essentially nothing in performance, and it will move the numbers in almost every case we run. I would very much like to be transparent about that in the release notes rather than have someone discover it for us in six months.

So I would flip your milestones around: capture reference arrays at the v1.0.10 tag first, diff dev against them, and treat live branch-vs-branch A/B as the second mode built on the same compare layer. Most of the machinery is shared, and the reference mode is the one that pays off before v1.1 ships.

Precision: I think the accuracy tier should run in float64
Our default precision is float32. Over a few hundred slices, two mathematically equivalent orderings — chunked versus monolithic potential building, a fused detection FFT, a different reduction order on the GPU — will differ at the 1e-6 to 1e-5 relative level with nothing whatsoever being wrong. That means the rtol = 1e-5 you propose for float32 sits right at the noise floor rather than comfortably above it, and a check that fires on ordinary floating-point reassociation will be muted within a month of being turned on.

I would rather the accuracy tier ran at precision: float64 and compared bit-exactly. "Bit-identical across seventeen workloads" is then a statement with real content, and anything that is not bit-identical is a genuine algorithmic change that deserves a sentence in the changelog. The float32 path would get a separate, looser check of its own: does it still track its own float64 result about as closely as it did before? That question is meaningful, whereas "did float32 change in the seventh digit" is not.

Answers to your six questions

1. Goals, non-goals, and whether drift should block. I agree with all of it as written. On blocking, I think both extremes fail: report-only gets skimmed and ignored, while a hard block means the propagator change cannot be merged at all. What I would like instead is that drift beyond tolerance fails the check, and that the failure is cleared by adding the case and a one-line reason to an accepted_changes file in the same pull request. The escape hatch is cheap enough that nobody is tempted to disable the check, the acceptance is recorded where a reviewer sees it, and as you already noted, that file is the changelog entry we would otherwise have to write by hand.

2. Which workloads. Your list covers the standard ones well. Looking at what is actually in flight in the repository, I would add a few:

Chunked potential building. The potential.slice-chunk-size default is "auto" and the auto value depends on available memory, so this needs both pinning in the preset and a case of its own.
CrystalPotential with frozen phonons. We have had real correctness bugs there, and the pool and tiling parameters change the result and not only the runtime.
A consistency pair of single-GPU against the dask.multi-gpu path with one worker. If those two ever disagree I would like to hear about it the same week rather than the same year.
Energy ensembles and focal spread. They are cheap to run and they exercise the distribution-weighting path, which we already know is inconsistent.
Non-orthogonal grids from #282, once that merges. It is an entire grid class that every caching layer has to respect correctly.
Do keep the multi-detector scan case prominent. The detector-side caching we added recently — the shared diffraction-pattern FFT and the detector bin cache — is exactly the kind of optimisation that can silently change results.

One practical correction about the CI tier: GPAW is not pip-installable for us. The abtem[gpaw] extra only pulls in hankel and sympy; GPAW itself comes from conda. So the GPAW core-loss cases cannot run on hosted runners. Use the built-in transition potentials for the CI tier and keep the GPAW cases for the local and standard tiers.

3. Tolerances. I would avoid a single relative tolerance. A CBED pattern spans six or more decades, and relative error evaluated on its noise floor tells us nothing except that the noise floor is noisy. We already have the right primitive in test/utils.py, where array_is_close takes check_above_abs and check_above_rel thresholds so that the relative comparison only looks at elements that carry real intensity. I would reuse that, and I would report a small vector per case rather than a single boolean: the maximum absolute difference normalised to the maximum of the reference, the relative error restricted to elements above roughly 1e-6 of the maximum, the total integrated intensity, and whether the arrays are bit-identical. The integrated intensity matters because it is the physically conserved quantity, and it is the one that would have caught the Diffraction code output being inconsistent depending on version of abTEM used? #261 problem directly.

For the frozen-phonon cases, please also assert that the seed is consumed in the same way, not just that it is set. Otherwise a refactor of the sampling order will present itself as a change in physics.

4. Where it lives — and here I want to split your question in two. The harness and the case definitions should be in the main repository, exactly as you propose, because they track the API of the code they test and CI needs them in-tree.

The reference arrays, however, cannot live in the main repository, and they cannot live in the docs repository either. The main repository is already around 160 MB and the docs repository is over 500 MB, and reference data accumulates along three axes at once: per release tag, per case, per device. Carrying that in either repository means every clone pays for it forever, and I do not want to do that to contributors. We already have a cautionary example in-tree: test/data/silicon_diffraction_patterns.zarr is a single 9.9 MB blob, and the test that used it has been commented out since the 1.0 line (test/test_measure.py:808). It is dead weight in every clone and it is not even buying us a test.

So I would repurpose abtem-benchmarks for this. It is still only about 1 MB, it is in the same organisation, and it already exists for benchmark-adjacent material, so it is a natural home. The harness would fetch a reference bundle for a given tag on demand and cache it locally. If the bundles turn out to grow faster than we expect, attaching them as release assets in that repository rather than committing them keeps the git history clean while leaving the URLs stable. A new dedicated repository would also be fine if you would rather keep the scaling studies undisturbed; I have no strong preference between those two, only that it is not the main repository.

Two smaller asks on the in-tree part. Make it a properly importable package so that python -m abtem_bench actually works, and put it under ruff and mypy so it does not rot the way the one-off scripts did. And because the cases come from the invoking checkout, please record a hash of the case definitions in every result file — two reports are only comparable if the case code behind them was the same, and that is the kind of thing we will otherwise discover the hard way.

Also, please do not delete the one-off scripts until the case each one encodes has actually been reproduced in the new harness. benchmark_potential_chunking.py is over 1,500 lines and a fair amount of that is accumulated tuning knowledge rather than a timing loop. It is worth mining before it is removed.

5. CI. A label-triggered quick tier on hosted runners with speed and memory confined to our own hardware is the right split; timing numbers from shared runners would be worthless anyway. On making the accuracy check required, I would say yes, but not at the start. Let the self-check demonstrate a zero false-positive rate over a month or so of real pull requests first. A required check that flickers gets switched off in irritation and never comes back on.

6. Build versus adopt. I agree with the custom harness, and your reasons are the right ones — asv cannot compare arrays and cannot do what we need with --python=same, and pytest-benchmark only times in-process. Just please do not rewrite the comparison helpers that test/utils.py already provides.

One last thing worth pinning
The accuracy preset should pin and then record the fully resolved configuration, because a changed config default is by far the most likely cause of an unexplained drift, and we have been changing them. The ones I would explicitly pin are precision, fft, fftw.planning_effort (the default is FFTW_MEASURE with a wisdom cache, so your suggestion of FFTW_ESTIMATE is right), dask.chunk-size, potential.slice-chunk-size, grid.round-to-fast-fft, and cupy.fft-cache-size — the last of which defaults to 25% of device memory, so it silently depends on which card is in the machine. Dumping the resolved config dictionary into the JSON sidecar costs nothing and will save somebody an afternoon eventually.

Please go ahead with the skeleton. If you start from the reference-array mode and the three cases that would feed the v1.1 release notes, it becomes useful to us immediately rather than at the end of the fourth milestone.

Thanks again,
Toma

🤖 Drafted with Claude Code, reviewed and edited by Toma.

[pzeiger]

Thanks Toma, this is exactly the input needed. The design has been revised along your lines and the full version is in the abtem-benchmarks repository under design/DESIGN.md, with one document per milestone next to it (M0 to M4). Below is the outline, so that you (or an agent) can review the design before any harness code is written.

Ordering: reference mode first

Agreed, and the milestones are flipped. M1 delivers capture at a tag and compare against that bundle; live branch-vs-branch A/B is M2 and reuses the same store and compare layer unchanged. The M1 output is the v1.1 drift table: dev against v1.0.10 for potential.infinite, hrtem.exitwave (exit wave plus SAED), stem.multidetector (BF, ADF, segmented) and diffraction.cbed, quick tier, CPU, float64.

One detail that makes the release-notes table attributable: algorithm=FourierMultislice(order=1) is accepted through **multislice_func_kwargs on both v1.0.10 and dev, so every M1 case carries an [order1] variant. Comparing dev[order1] against the v1.0.10 default isolates everything that is not the propagator. If that comparison is bit-identical, the release note can say so; if it is not, the residual is a finding that goes into the same report.

v1.0.10 is the tip of main and already carries the numpy >= 2, zarr >= 3.1 and dask pins that dev has, so the reference capture runs in the shared environment with a plain worktree. Only v1.0.9 and older need the isolated-venv mode, which is scheduled for M3.

Precision and tolerances

Agreed on both. The accuracy preset runs at precision: float64 with FFTW_ESTIMATE, one FFTW thread, the synchronous scheduler, explicit gpts, batch and chunk sizes, fixed seeds and pinned thread environment, and the expectation on CPU is bit-identical. A self-check (same ref captured twice, interleaved) must come back IDENTICAL before any reference is captured, and it doubles as the false-positive monitor. On GPU, cuFFT and CuPy reductions are not order-stable and the finite-projection integrator is already known to differ at about 1e-7 between runs, so GPU float64 is compared at the floor the self-check measures on that machine, with the identical flag reported but not required.

The float32 path gets the separate check you describe: each case also runs at float32 in the same bundle, the metric is its distance from its own float64 result, and compare flags only when that distance grows relative to the reference bundle.

The per-output comparison is the vector, not a boolean: bit-identical flag, max absolute difference over max of the reference, relative error restricted to elements above 1e-6 of the maximum, integrated intensity, and exact equality of shape, dtype and axes. array_is_close is reused rather than rewritten: it moves to abtem/core/testing.py unchanged (with test/utils.py re-exporting it, so the test suite does not change) and a close_stats() returning the vector is added next to it. Both its tolerances default to infinity, so the harness always passes them explicitly.

Frozen-phonon cases emit the consumed random state as an output (displacements and the seed tuple) alongside the image, so a refactor of the sampling order shows up as a result change rather than as physics. CrystalPotential with num_frozen_phonons does the same with its per-member pool seeds.

Drift policy

Adopted as you describe: drift beyond tolerance fails the check and is cleared by an entry in benchmarks/accepted_changes.toml in the same pull request (case glob, reference tag, one-line reason, PR number). The report renders matched entries as the changelog table with the measured drift vector next to each, entries that no longer match anything are reported as stale, and unknown case globs fail validation, so the file cannot rot.

Cases

All of your additions are in the matrix. Chunked potential building is its own case with explicit chunk sizes (1, 5, all) whose outputs must agree with each other, and potential.slice-chunk-size is pinned in the preset. CrystalPotential with frozen phonons covers pool sizes and tiling. Energy ensembles and focal spread are two cases and include both ensemble_mean settings. The single-GPU versus dask.multi-gpu one-worker pair is a guarded case that runs where dask-cuda and two devices are visible (Perlmutter), since the auto-start requires more than one visible device. Non-orthogonal grids are listed for when #282 merges. The multi-detector scan is one of the four M1 cases.

Two more cases come from the issue history rather than from the API list. consistency.cache_reuse builds Si at one grid, Au at another, then Si at the first grid again, in one process, on CPU then GPU: the cache-keying bugs of the last months (integral tables and scattering factors keyed on the symbol only, device caches keyed on the string "gpu") are only visible when two grids or two devices share a process, and a suite that runs one shape per subprocess would never see them. grid.fastfft_pair keeps the 2623×2271 versus 2625×2304 pair on the tWSe2 cell that #347 was measured on.

One correction to the CI point: abTEM has no built-in transition potentials. Both radial solvers in core_loss.py import GPAW unconditionally, and xc=None does not bypass that. What exists is the array-injection path, TransitionPotentialArray(Z, array, energy, extent), on both refs. The plan is therefore: the CI tier runs coreloss.image on a seeded synthetic transition potential (the same construction test/utils.py already uses), which exercises the scatter and detection machinery deterministically but carries no physics, and the local tiers additionally use a real O K edge computed once with GPAW and stored as a small asset in abtem-benchmarks. Reports label the two explicitly.

Where things live, and the two smaller asks

Harness and cases in the main repository under benchmarks/, as a real package with its own pyproject.toml (abtem_bench, console script abtem-bench, python -m abtem_bench works with PYTHONPATH=benchmarks or after an editable install), excluded from the wheel, under ruff (which already covers benchmarks/) and mypy, with its own unit tests in the normal test job.

One small proposal on naming: the repository is abtem-benchmarks, while the organisation's other repositories are abTEM, abTEM-legacy and doc. Renaming it to abTEM-benchmarks would follow the existing convention; GitHub redirects the old name, so nothing breaks, and it needs admin rights on the repository, which only you have.

Reference bundles go to abtem-benchmarks as release assets (refs-v1.0.10, one asset per tier, device, preset and machine fingerprint), with an index file in git and a refs fetch command that caches locally. That keeps the git history of that repository at its current size regardless of how many bundles accumulate. The design documents and the frozen legacy scripts live there too.

Every manifest records a sha256 over the case sources and the harness version; compare refuses to pair bundles with different hashes unless told to, and says so in the report. The fully resolved abtem.config.config, the dask config and the thread environment are dumped into every manifest, and the accuracy preset pins the keys you list plus cupy.fft-cache-entries, fftw.threads, mkl.threads, dask.lazy and the progress bar.

On the legacy scripts: nothing is deleted until M4, and only per reproduced case. All 13 have been copied to abtem-benchmarks under legacy/ with a provenance file that records, per script, what it measures, which tuning knowledge it holds and which case reproduces it. benchmark_potential_chunking.py is the main source for the meters, the error classification (including CUDA_ERROR_ILLEGAL_ADDRESS counting as OOM) and the size tables of the large tier.

Measurement details worth stating

Peak host memory comes from os.wait4 on the case subprocess, not from a sampler, because a sampler undercounts short spikes and a polling instrument once tipped a memory-pressured run into a livelock. Memory is measured one shot per process and the median over several processes is reported, because the GPU path retains about 40 MB of native memory per identical repeat and FFTW_MEASURE trial plans produced a 2.7× run-to-run spread on unchanged code. Cold-start time is recorded separately from warm repeats, and whether warm-up applies is a per-case declaration rather than a global rule. For lazy cases an optional meter records the task count and the pickled size of the graph, which is the axis that #386 moved. Distributed cases read worker memory from scheduler_info() only.

CI

As agreed: label-triggered, quick tier, CPU, float64, PR head against its merge base in one job, bit-identical expected, accepted_changes.toml as the escape hatch. A second, purely informational job compares the PR head against the latest tag bundle so the v1.1 table is visible on every PR. The required-check proposal comes after about a month of self-check data with zero false positives. Speed and memory are judged only on our own hardware, using the same command line.

Milestones

M0 (done): consolidation of design, legacy scripts and the reference-bundle home in abtem-benchmarks. M1: reference mode, harness core, the four v1.1 cases, self-check, v1.0.10 quick-tier CPU bundle, the dev versus v1.0.10 report with [order1] attribution, one PR. M2: live A/B, consistency pairs, float32 tracking, graph meter, the full matrix, the legacy mining table. M3: release-asset hosting, isolated mode for old tags, memory mode, large tier and the A100 bundle, Perlmutter runbook. M4: the workflow, the required-check decision, docs, legacy deletion PR.

If the outline reads right, M1 starts now and the first thing to appear here will be the dev versus v1.0.10 table.

🤖 Written by Claude Code — Paul reviewed and posted it

[pzeiger]

A follow-up on where the non-code parts should live, because rereading the design raised a doubt worth settling before M1 rather than after.

With the harness and the cases in the main repository and the reference bundles delivered as release assets, the question is what abtem-benchmarks is actually for. Everything currently placed there could live in abTEM/abTEM instead: assets can be attached to the existing release of a tag just as well (git history stays untouched either way), the design documents track the harness code and arguably belong beside it under benchmarks/docs/, and frozen copies of the legacy scripts are redundant, since git history keeps them forever and a provenance file can cite git show <sha>:benchmarks/<file>.

Two things speak against putting the bundles on abTEM releases. Bundles are captured after the tag by a later harness and get re-captured whenever the case matrix changes, so one tag would accumulate several generations of multi-hundred-MB assets on the page users visit to download abTEM. And publish.yml runs the PyPI publish on every release published event, so a separate "refs" release must never be created there; attaching to existing releases is the only safe form. Both are manageable, neither is decisive.

What the main repository cannot hold well is a results history. Once the suite runs weekly on our own hardware it produces per-commit speed, memory and self-check records, a few KB of JSON each, that do not map to releases and grow with every run. That series is what answers "when did this get slower" and what the required-check decision will be based on, and it is why asv keeps a separate results repository. The 2023 scaling-study notebooks already in abtem-benchmarks are the same kind of content.

Options, then:

1. abtem-benchmarks as the data and history repository: reference bundles as its release assets, committed per-commit results and self-check history, archived reports, the scaling studies. Design docs and provenance move into abTEM/benchmarks/docs/; the legacy copies are dropped in favour of SHA citations. Pro: a clear purpose, no growth in the main repository, trend questions answerable. Con: two repositories to keep in step, and a second place contributors have to know about.
2. Everything in abTEM/abTEM: bundles attached to existing releases, history kept only in CI job summaries, abtem-benchmarks left as the 2023 studies. Pro: one repository. Con: release pages carry benchmark artefacts, trend questions across commits become hard to answer, and the publish.yml constraint has to be remembered by everyone who ever uploads a bundle.

Option 1 seems the better fit, but this is a maintainer's call. The consolidation done so far (design docs, legacy copies) is easy to move either way, so nothing is blocked; M1 only needs to know where refs fetch should look.

🤖 Written by Claude Code — Paul reviewed and posted it

[pzeiger]

A second follow-up, on what the case matrix reaches below the workflow level, and on how a regression there actually gets found in the first place.

#438, open now, is a good concrete test of the design, and it fails on both counts. It was found by investigating low GPU utilization on a production core-loss EELS scan running on a single A100: nvidia-smi showed about 99% GPU utilization alongside only 100 to 180 W of a 400 W power cap, and a full time breakdown of that scan placed 87.3% of its wall-clock inside one kernel call, sum_run_length_encoded in abtem/core/_cuda.py — the hand-written CUDA kernel behind every radial and polar detector's binning step. It launched one thread per bin and had that thread loop serially over the batch axis and every pixel in the bin's segment, so cost scaled with batch size regardless of how many streaming multiprocessors the GPU had. On the exact workload that surfaced it, the same 6733 calls into this kernel dropped from 715.70 s (87.3% of wall-clock) to 2.36 s (2.2%) after the fix, a 303x reduction in that one step.

The kernel-level numbers in the PR show why that is batch-dependent rather than a flat cost: on the A100 the scan ran on, 5.1x at a small batch (10 rows) against 399.5x and 442.6x at the batch sizes a real FlexibleAnnularDetector call produces (80 and 1,000 rows). That range matters for the case matrix specifically: the multi-detector case already sketched in the design uses max_batch=8 at the quick tier, close to the shape that only shows 5.1x. A quick-tier case would likely see a real but far smaller effect than the one that motivated the fix, and would report it as one workflow's total time moving, not as sum_run_length_encoded regressing.

That is the first gap: attribution. A regression in stem.multidetector's total time says the workflow got slower, not which of the routines it calls is responsible, and at quick-tier batch sizes the effect may be too diluted to flag at all. The second is cost: a kernel call here costs microseconds to milliseconds, so sweeping batch size from 10 to 1,000 to expose the effect is affordable for the kernel alone and is not for the workflow it sits inside.

The natural next question is why not build that same time breakdown into the workflow cases directly, timing potential build, multislice and detection as named phases of run(). Two reasons say not to. It would perturb what it measures: getting a clean per-phase number this way needs a synchronization point at each monkeypatched boundary, which is exactly what #438's own investigation did, running eager (lazy=False) to get it. That is a reasonable choice for a one-off diagnostic against a workload already known to be the problem, and a bad one for the number the suite uses to decide whether a commit regressed — the case matrix already plans lazy and multi-GPU cases (stem.haadf's eager/lazy variants, the single-GPU-versus-dask.multi-gpu pair), and forcing the same synchronization into every quick-tier run of those would change how the scheduler overlaps work, not just measure it. The design already holds its meters to a stricter bar than that: the VRAM meter is deliberately limited to driver and allocator queries that "run no code on the computation," and the graph-transport meter is deliberately CPU-only and reads the built graph before compute rather than during it — both chosen so the meter cannot be the thing that changes the answer. And phase boundaries do not survive legitimate refactors: #366 fused what used to be one FFT call per detector into a single shared call, exactly the kind of change that would silently invalidate a hardcoded "detection phase" boundary, with nothing wrong having happened at all.

Lacking anywhere better to put it, the fix's own test file shows the cost of that gap. test/test_radial_bin_sum_kernel.py adds a correctness sweep against a CPU oracle, which is exactly where it belongs, but alongside it a hardcoded absolute-time assertion, elapsed < 0.05, because there is currently no comparative alternative. The bound is deliberately loose, an order of magnitude above the post-fix measurement and well below the pre-fix one, precisely so it does not need pinning to one machine's numbers. That looseness is also its limit: it only catches a regression back to roughly the old serial-batch cost, not a more moderate one, and it carries no history to show whether the margin is shrinking.

This is not a one-off kernel, either. abtem/core/_cuda.py carries a second RawKernel (_INTERPOLATE_RADIAL_KERNEL), and abtem/finite_difference.py has its own hand-written stencil kernel. Hand-written CUDA is a recurring category of code in this codebase, not a single hot spot.

Proposal: keep the workflow cases exactly as designed, unperturbed, as the signal that something changed — and add two more mechanisms underneath them that answer the questions a workflow case cannot.

The first is a profiling mode, not a case: abtem-bench profile <case-id>, which generalizes what the #438 investigation did by hand into a reusable harness command. It runs one case once, instruments its major internal calls with the same monkeypatch-and-synchronize approach, and reports the percent of wall time in each. It is never compared across refs and carries no tolerance or verdict — it answers "where," on demand, once a workflow case or an operator's own nvidia-smi says "something." For a lazy or multi-GPU case it may need to force eager execution and pin the run to a single worker to get clean numbers, and says so plainly rather than pretending the result represents ordinary usage.

The second is a routine.* case category: cases that target one internal function directly rather than a public workflow. routine.radial_bin_sum, sweeping batch size and bin count, is the case the profiling mode's finding earns, and would be the first entry. Targeting a private function has a real consequence: internal signatures change far more often than public ones, so requires= guards are load-bearing routinely rather than occasionally, and a case can legitimately go UNSUPPORTED after a refactor that changed nothing observable. It is earned by an incident, not written speculatively, the same bar the legacy scripts had to clear to be worth keeping — this is deliberately not "benchmark every kernel," which is how the scripts now being consolidated accumulated in the first place. Ownership follows from targeting internals: whoever touches a hot path enough to need a routine case adds or updates it in the same pull request, the way a unit test is expected today, rather than the harness maintaining a central routine matrix on its own schedule. And it measures speed only — correctness stays in test/, against the same CPU oracle #438 already uses, so the two kinds of test do not duplicate each other's job.

Neither addition changes the earlier question about the two repositories: the profiling mode and the routine cases are both code, so they sit with the harness in abTEM/abTEM, under the same ownership rule as the workflow cases. The routine tier adds weight to keeping a results history somewhere, since "did this kernel regress since last month" is exactly the kind of question a point-in-time comparison cannot answer and a time series can.

Both additions revise the design as posted: the profiling mode is a new component with no analogue in it, and the routine tier changes a line in the non-goals list. That seemed worth raising here before either goes into the design document rather than folding them in unilaterally. If the split looks right, routine.radial_bin_sum becomes part of the M2 case matrix and the profiling mode becomes part of the CLI alongside list, capture and compare; if the ownership model or the profiling mode's eager-only limitation looks wrong, or the whole idea still reads as scope creep from where you sit, that is useful to know now.

🤖 Written by Claude Code — Paul reviewed and posted it