ENH: Add performance benchmark CI builds

[thewtex]

No description provided.

[jhlegarreta]

Speechless. Awesome.

[kwrobot-v1]

Errors:

• Failed to run the checks: service error.

[thewtex]

Errors:

* Failed to run the checks: `service error`.

CC @bradking @mathstuf

[mathstuf]

Hmm. Seems we got a timeout on one of our queries.

Do: check

[jhlegarreta]

Updated macOS version from 10.13:
c93e0c6#diff-c8d4e43125ff7fbe0d808694e0b2ab37de80808b1d9a54fed631330117d3cd4bR90

to macOS 11, and Python from 3.7 to 3.9:
c93e0c6#diff-c8d4e43125ff7fbe0d808694e0b2ab37de80808b1d9a54fed631330117d3cd4bR104

Preferred to avoid style changes to the existing parts, e.g.:
c93e0c6#diff-c8d4e43125ff7fbe0d808694e0b2ab37de80808b1d9a54fed631330117d3cd4bR8

Not sure if this will be enough to have this working.

If Azure allows to re-use workflows, cleanest would be to relocate the performance benchmarking part to a separate file so that the macOS build part can report its status regardless of the benchmarking part.

[jhlegarreta]

So the benchmarks add 22 minutes;
https://dev.azure.com/itkrobotmacos/ITK.macOS/_build/results?buildId=11489&view=logs&j=68e22afd-607b-5133-9236-337c135f0832

If this is deemed excessive, they could be triggered e.g. once or twice a week.

The next thing will be how to display those reports on a website.

[thewtex]

I started an ASV port on the ITKPerformanceBenchmarking repository. Eventually we should be able to run asv compare similar to what @yarikoptic did with datalad.

[yarikoptic]

FWIW we ran into some minor issues with recent asv and our setups, just in case you do too -- might want to restrict for now to < 0.6.2 and make sure that you have asv[virtualenv] in your setup*, as we did in https://github.com/con/fscacher/pull/91/files#diff-fa602a8a75dc9dcc92261bac5f533c2a85e34fcceaff63b3a3a81d9acde2fc52R49 which is yet another project of ours which does similar benchmarking on CI

[hjmjohnson]

@thewtex — the rebase + origin/master → origin/main workaround in .github/workflows/perf-benchmark.yml (commit c4e39a64) got the asv base vs head check green (11m59s). All Azure C++ pipelines pass on the fresh tip c4e39a64; remaining pending are the usual long-runners (Python-wrapping + Pixi-Cxx).

To remove the in-workflow patch permanently:

1. Merge InsightSoftwareConsortium/ITKPerformanceBenchmarking#113 (2-line fix to asv.conf.json + README-asv.md).
2. Bump Modules/Remote/PerformanceBenchmarking.remote.cmake GIT_TAG here to that merge SHA.
3. Delete the python -c sed block from the Stage ASV harness step.

[hjmjohnson]

@thewtex — ITKPerformanceBenchmarking#113 merged (b24324f922). Force-pushed tip 2359970c24 that (1) bumps Modules/Remote/PerformanceBenchmarking.remote.cmake GIT_TAG to the merge SHA and (2) drops the python -c sed-patch from the Stage ASV harness step — the workflow is now back to your original minimal form.

Net diff vs the pre-rebase tip cfa9a3ff0d:

• Modules/Remote/PerformanceBenchmarking.remote.cmake +1/-1 (pin bump)
• rebased onto current upstream/main (was 11 commits behind, now 0)

Fresh CI kicked off; should settle similarly to the previous run (asv was 11m59s last round).

[greptile-apps]

Greptile Summary

This PR adds a new GitHub Actions workflow (perf-benchmark.yml) that runs ASV (airspeed velocity) benchmarks comparing the PR HEAD against its merge-base, and bumps the PerformanceBenchmarking remote module pin to a commit that introduces the required ASV harness.

• P1: The workflow_dispatch trigger causes the "Resolve merge-base and HEAD SHAs" step to fail — github.base_ref is empty outside of pull_request events, making git fetch origin "" fatal.
• P2: CMakeVersion: "4.0.1" and ExternalDataVersion: 5.4.5 are declared in env but never referenced; if CMake 4.x is genuinely required, an installation step is missing.

Confidence Score: 4/5

Safe to merge after fixing the workflow_dispatch / github.base_ref breakage.

One P1 defect: manually triggered runs fail immediately due to an empty github.base_ref. The cmake pin bump is clean and the overall workflow design is sound.

.github/workflows/perf-benchmark.yml — the workflow_dispatch + github.base_ref interaction and the unused env vars.

Important Files Changed

Filename	Overview
.github/workflows/perf-benchmark.yml	New CI workflow running ASV benchmarks on base vs HEAD; broken for workflow_dispatch due to empty github.base_ref, and has unused CMakeVersion/ExternalDataVersion env vars.
Modules/Remote/PerformanceBenchmarking.remote.cmake	Single-line GIT_TAG bump to pin the commit that introduced the ASV harness (asv.conf.json) required by the new workflow.

Sequence Diagram

sequenceDiagram
    participant GHA as GitHub Actions
    participant Base as itk-base (merge-base)
    participant Head as itk-head (PR HEAD)
    participant ASV as ASV Harness

    GHA->>Base: checkout @ merge-base
    GHA->>Head: checkout @ PR HEAD (full history)
    GHA->>Base: cmake configure + build benchmark targets
    GHA->>Head: cmake configure + build benchmark targets
    Note over Head: PerformanceBenchmarking module populated with asv.conf.json
    GHA->>ASV: pip install harness (from itk-head)
    GHA->>ASV: asv machine --yes
    GHA->>ASV: asv run --quick @ merge-base SHA
    Note over ASV: continue-on-error: true
    GHA->>ASV: asv run --quick @ HEAD SHA
    GHA->>ASV: asv compare merge-base HEAD
    ASV-->>GHA: asv-compare.txt + results artifact

Loading

_{Reviews (1): Last reviewed commit: "COMP: Bump PerformanceBenchmarking pin t..." | Re-trigger Greptile}

[greptile-apps]

github.base_ref is empty for workflow_dispatch

When the workflow is triggered manually via workflow_dispatch, github.base_ref is an empty string. Line 58 expands to git fetch origin "", which fails immediately with fatal: couldn't find remote ref. The entire asv-compare job becomes broken for manual runs.

You either need to guard this block behind a pull_request-only condition or supply a fallback (e.g. ${{ github.base_ref || 'main' }}):

Suggested change

	HEAD_SHA=$(git rev-parse HEAD)
	echo "merge_base=${MERGE_BASE}" >> "$GITHUB_OUTPUT"
	echo "head=${HEAD_SHA}" >> "$GITHUB_OUTPUT"
	echo "merge_base=${MERGE_BASE} head=${HEAD_SHA}"
	- name: Checkout ITK at merge-base (shallow)
	uses: actions/checkout@v4
	git fetch origin "${{ github.base_ref || 'main' }}"
	MERGE_BASE=$(git merge-base HEAD "origin/${{ github.base_ref || 'main' }}")

[greptile-apps]

Declared env vars are never consumed

ExternalDataVersion: 5.4.5 and CMakeVersion: "4.0.1" are defined at the workflow level but referenced nowhere in any run step. CMakeVersion is particularly suspect: Ubuntu 24.04's default apt ships CMake 3.28, not 4.0.1. If CMake 4.x is required (e.g. for FetchContent changes in 4.0), a step to install it via pip install cmake==4.0.1 or the jwlawson/actions-setup-cmake action is missing. If they're not needed, removing them avoids confusion.

[greptile-apps]

--quick produces single-shot, noisy measurements

asv run --quick executes each benchmark exactly once. With no statistical averaging, the compare output will reflect OS scheduling noise as much as real code differences, making it hard to distinguish genuine regressions from measurement jitter. Consider omitting --quick (or replacing with --min-runs 3) so ASV collects a small sample and reports median-based results, which is the standard for reliable CI benchmarking.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!