Benchmark Signal Policy: weekly benchmark reporting

[AnshulPatil2005]

ref : opencax/GSoC#114
I wanted feedback on a proposed Benchmark Signal Policy for Manifold benchmarks @elalish @pca006132.

Current PR guard (already in #1680)

• Fixed repeats via PERF_REPEATS (3 currently).
• Base vs head on the same runner.
• Mean-based per-benchmark comparison.
• Dual-threshold warning:
  • delta_pct >= REGRESSION_WARN_PCT (20)
  • delta_ms >= REGRESSION_WARN_ABS_MS (10)
• Non-blocking warnings.

Proposal

• Keep PR guard behavior unchanged.
• Add weekly scheduled base-only runs for trend/history.
• Keep PR + weekly outputs on one schema (aligned with current guard).

Shared schema(result.json)

• Per-run sample: benchmark, run_index, time_sec
• Per-benchmark aggregate: mean_sec, median_sec, stdev_sec, n_runs
• Metadata: commit_sha, workflow, runner, os, compiler, timestamp, schema_version CPU model + logical core count (lscpu)
  Memory total (free -h)

Later all this data can be fed into a dashboard(frontend)

useful additions/follow ups(not confirmed) :
baseline_ref in JSON (which commit/run it compares against).

Trend fields (week_over_week_delta_pct, 4w_median).

Optional rerun-on-noise rule (retry once if variance exceeds threshold).

Lightweight weekly report comment/discussion post automation.

[elalish]

I would skip REGRESSION_WARN_ABS_MS, since different tests will likely be quite different in timing. Let's just make sure each test takes long enough that it can give reasonable timings, say 1-10 seconds.

How many runs do you expect to aggregate? Median and stdev are really only useful if your sample size is >>10. I think we might be better off with min, max, and mean.

[AnshulPatil2005]

I was considering benchmark-specific REGRESSION_WARN_ABS_MS values, but I agree that won’t be practical to maintain long-term.

for the current plan i am thinking 5 runs maybe so i will stick with min,max and mean in that case

[AnshulPatil2005]

Possible benchmarks to test for weekly benchmarks
some of the regressions was later treated as fixed.

1. perfTest size sweep (extras/perf_test.cpp)
   Source: PR benchmark guard (base vs head) #1676

2. Sequential union chain stress (m += part repeated)
   Source: accidental quadratic runtime in csg tree #989

3. Batch union stress (BatchBoolean on same set as basic travis config #2)
   Source: Boolean vs BatchBoolean #1579

4. Thin-triangle BVH stress boolean (long-thin vs regular triangulation)
   Source: Massive performance drop caused by long thin triangles #1156

5. Triangulator many-holes stress (keyhole-heavy polygon cases)
   Source: Triangulator poor performance when the polygon contain many holes #1158

6. MeshGL -> Manifold import pipeline stress (large mesh construction path)
   Source: Improve performance of MeshGL -> Manifold construction #1138

[AnshulPatil2005]

For these cases, should I first isolate the actual slow path in each one, and then reduce that into a small reproducible path for the weekly benchmark set?

Also, once that path is identified, what would be the preferred way to turn it into a specific benchmark test here? Is there an existing pattern in the repo that I should follow for that?

[pca006132]

Just add those regression tests to the benchmark set should be good enough. I think they are there, no?

[AnshulPatil2005]

#1 already exists directly as a benchmark as far as i know, for the rest of them I'll take a look if they exist.

[AnshulPatil2005]

For 2, 3, 5, 6, the relevant tests seem to exist in the repo(although it has partial coverage), so instead of creating new cases from scratch I can reuse those i guess ?

[pca006132]

Yup, and I think you dont need to minimize them, just use them as-is should be fine.

[AnshulPatil2005]

I ran a local single-thread discovery pass over the 1000 Ember cases and checked the phase timings.

Top 3 Boolean-heavy cases:

1. Case 668 (472042 vs 99865) -> boolean = 2720.7 ms
2. Case 696 (1147240 vs 520644) -> boolean = 1220.7 ms
3. Case 17 (88566 vs 58011) -> boolean = 1040.7 ms

Top 3 to manifold heavy cases:

1. Case 404 (441714 vs 69685) -> to manifold = 201.2 ms
2. Case 842 (38562 vs 368147) -> to manifold = 191.9 ms
3. Case 542 (1063861 vs 75121) -> to manifold = 170.1 ms

ember-benchmark-1.csv

[AnshulPatil2005]

I mean your raw csv file is wrong: your total time double counted those intersection phases. This makes the normalized timing wrong as well.

Oh, you are right.
the calculations for norm are inflated in that case; I'll regenerate the table.

[AnshulPatil2005]

corrected raw csv ->phase_timingsnew.csv

filtered for at least one independent raw phase timing is > 300 ms: ->

case_index	full_phase_sum_ms	Assembly_ms	Triangulation_ms	Simplification_ms	Sorting_ms	Intersect12 P->Q_ms	Intersect12 Q->P_ms	Winding03 P_ms	Winding03 Q_ms	Assembly_norm	Triangulation_norm	Simplification_norm	Sorting_norm	Intersect12 P->Q_norm	Intersect12 Q->P_norm	Winding03 P_norm	Winding03 Q_norm
16	661	5	21	4	1	244	385	0	1	0.007564	0.031770	0.006051	0.001513	0.369138	0.582451	0.000000	0.001513
84	621	14	27	16	11	333	214	5	1	0.022544	0.043478	0.025765	0.017713	0.536232	0.344605	0.008052	0.001610
667	1759	11	15	6	3	679	1042	0	3	0.006254	0.008528	0.003411	0.001706	0.386015	0.592382	0.000000	0.001706
695	796	18	25	25	19	481	218	10	0	0.022613	0.031407	0.031407	0.023869	0.604271	0.273869	0.012563	0.000000

[pca006132]

maybe make the filter just >10ms? do we have cases where intersect12 is not taking over 80% of the time?

[AnshulPatil2005]

there are 257 rows with >10ms filter and from these 257 rows: 211 rows are not taking 80% time without >10ms filter that number would be 951

[AnshulPatil2005]

Some larger examples with Intersect12 <= 80%:
260
974
582
582 being the clearest example

case_index	Intersect12 share	full_phase_sum_ms	Triangulation_ms	Simplification_ms	Sorting_ms
260	0.742	310	25	21	12
406	0.743	261	18	20	15
551	0.617	248	25	27	23
202	0.726	241	18	16	13
974	0.750	220	18	14	9
46	0.684	212	22	18	10
288	0.689	190	16	16	13
531	0.629	186	21	19	14
596	0.681	182	17	15	12
582	0.464	151	42	29	3