Catching performance regressions on main: compare benchmark runs, and a nightly A/B I can run

[tae898]

Why. #10432 merged on Oct 20 and did what it set out to do: whole-graph reads over a Graph Analytical View with an overlay got about twice as fast (#10411). The same merge made point reads on the view about three times slower while writes are running (#10471: per-query time 2.96x, 95% CI 2.48 to 3.12, 7 of 7 interleaved pairs, against its parent commit). No test failed, because the tests check answers, not speed, and nothing compares the speed of one build with the previous one. I caught it only because I happened to A/B that merge the same night.

What exists today. benchmark-tests.yml runs the *Benchmark classes weekly and load-tests.yml runs daily, both on shared ubuntu-latest runners and with --fail-never. So the numbers are produced but not compared with anything, and shared runners are too noisy to see a change smaller than a large jump.

Proposal, in three small steps.

1. Keep each benchmark run's numbers (an artifact or a results branch) and compare them with the previous run. When a benchmark is slower beyond its usual spread, open an issue instead of failing the build.
2. Add benchmarks for the shapes that have regressed before. The one above is a small program (a view, a reader running point and whole-graph 2-hop counts, a writer committing at a fixed rate); I would contribute it as a *Benchmark class.
3. Measure the merges on a quiet machine. I already run interleaved A/B rounds with a noise model on a dedicated host, and from late November that host re-measures each new ArcadeDB version continuously. I can run a nightly A/B of main against the previous night and file what moves, with the statistics.

Would you take 1 and 2 as pull requests? And which workloads would you want covered first?

[tae898]

A data point for this proposal: my own sweep of current main against 26.10.2 (ab.py, interleaved runs, the same figures as #10471) caught another one this week. #10458 made Cypher variable-length hops ([*1..4] count(DISTINCT b)) about a third slower on JDK 25 and JDK 21, and it is still on main; bisected and filed as #10658. As with #10471, no test failed. I now run such a sweep before each release cut and after merges that touch a hot path, and will post anything it finds the same way; a comparison of benchmark runs in CI would catch these at the PR instead.

[robfrank]

I have something in mind performance regression testing based on load tests an maybe more.

[tae898]

Sounds good. Four things from my sweeps that should carry over to load tests: run the two builds interleaved (ABBA) on pinned cores, drop the first round, drop rounds that overlapped other load on the machine, and call a change only when the 95% CI of the paired ratio clears a threshold (I use 5% with at least 10 clean pairs). Without those, run-to-run noise on a laptop or a shared runner was as large as the regressions I found. When there is a first version, I can run it through the same A/B on my machines and compare what each catches.