feat(bench): a performance harness the published numbers come from

[FBumann]

docs/benchmarks.md published a table whose harness had been deleted — the file said so itself. The one quantitative claim the project makes could not be re-run. This adds bench/ and regenerates the numbers from a committed results file.

The harness

uv run python -m bench.run --sizes xs s m l --repeat 2 --memory-limits 256MB 1GB 4GB
uv run python -m bench.report bench/results/latest.jsonl

Two arms, same YAML, same output format: fk.build + write_lp against Model.to_file(io_api='lp-polars'). That is the only comparison worth making first — the linopy lane accepts exactly the same language (hard rule 3), so it is an oracle rather than a rival dialect.

Design, all of it forced by something already recorded in the doc:

• One process per measurement. Peak RSS is a property of a process; a second arm in the same interpreter inherits the first's high-water mark. bench/run.py never imports farkas or linopy.
• ru_maxrss, not a tracker — operational finding 4 says memray slows duckdb ~8x and overcounts polars.
• The parity gate runs first. Smallest rung of each case solved on both arms, objectives compared to 1e-9, run aborts on mismatch. The differential suite proves parity for the language, not for the data this harness generates.
• Import time is its own phase, excluded from wall_seconds — at the smallest rung linopy's import alone exceeds farkas's entire build.
• Failures are results, written to the JSONL with the exception that killed them, and rendered as cells. See below for why that mattered.

Two cases, chosen for the SQL shape they stress rather than for coverage: dispatch (pointwise + one sum) and transport (three group_sum joins per row).

What the run found

Parity gate passed with bit-identical objectives on both cases, so the timings describe the same model.

	wall	peak RSS
dispatch 10M vars	6.67 s vs 3.19 s (2.1x slower)	0.88 vs 2.19 GB (0.40x)
transport 9.8M vars	9.25 s vs 3.96 s (2.3x slower)	1.35 vs 1.88 GB (0.72x)
either case, <= 100k vars	0.24-0.85x — farkas faster	~0.7x

The memory win is real and widens with model size; the sub-10^5 win is just linopy's fixed overhead, not an engine property.

Two findings worth more than the table, both written up in docs/benchmarks.md and neither fixed here:

• Peak RSS does not track memory_limit (Peak RSS does not track memory_limit: hard rule 4 holds comparatively, not literally #144). Over a 16x budget range peak at 10M variables moves 0.77 -> 0.90 GB, at 2-3x the budget. Hard rule 4 holds comparatively, not literally.
• transport at 9.8M variables under 256MB OOMs rather than spilling (Constraint assembly OOMs instead of spilling when the group_sum fan-in is wide #145), in the unchunked constraint assembly — directly under the comment asserting that plain hash aggregates spill on their own. dispatch at 10M is fine, so the trigger is the group_sum fan-in.

Docs

docs/benchmarks.md now leads with generated results and a "what the numbers say" section that states the 2x build cost plainly. The earlier 107M-variable numbers are kept in a clearly labelled not reproducible section — nothing in the new ladder replaces that capability proof. Operational finding 6 records the OOM. ARCHITECTURE.md gains one line pointing at bench/ as how a claim here gets falsified.

bench/.cache/ is gitignored (regenerated deterministically); bench/results/ is tracked, because the provenance is the point.

🤖 Generated with Claude Code

Issues raised from this run

• Peak RSS does not track memory_limit: hard rule 4 holds comparatively, not literally #144 — peak RSS does not track memory_limit
• Constraint assembly OOMs instead of spilling when the group_sum fan-in is wide #145 — constraint assembly OOMs instead of spilling on wide group_sum fan-in
• Build is ~2x slower than the eager lane end to end; 71% of build is the label sort #146 — build is 6-10x slower than the eager lane, while the LP writer is already at parity
• Benchmark coverage: solver_direct, storage/MILP cases, sparsity sweep, speed-of-light floor #147 — benchmark coverage: solver_direct, storage/MILP, sparsity sweep, speed-of-light floor

[coderabbitai]

Warning

Review limit reached

@FBumann, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 47 seconds

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ea203438-75b6-478c-af4f-b302fc1dcd83

📥 Commits

Reviewing files that changed from the base of the PR and between bc6918c and 869bbd4.

📒 Files selected for processing (14)

• .gitignore
• ARCHITECTURE.md
• bench/README.md
• bench/__init__.py
• bench/_run_case.py
• bench/cases.py
• bench/models/dispatch.yaml
• bench/models/nodal.yaml
• bench/models/transport.yaml
• bench/report.py
• bench/results/latest.jsonl
• bench/run.py
• docs/benchmarks.md
• pyproject.toml

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch worktree-bench-harness

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[FBumann]

Pushed two changes: a real sparse case, and a fix to what the clock was actually measuring.

nodal replaces the abstract mask

dispatch declares where: p_max > 0 against a p_max that is always positive — its mask removes nothing, so the ladder had only ever measured a fully dense coordinate product. The report now has a live column, so that reads as 100% rather than being a trap, and the vacuous mask stays deliberately: paying for a mask that removes nothing is itself a measurement.

The new case is sparsity as it actually occurs. Dispatch over (snapshot, node, tech) where a technology only generates at a node it is installed at — no offshore wind inland, no hydro without a river. PyPSA attaches generators to buses, Calliope declares techs at nodes; in YAML it is a where over the capacity table. 50 nodes x 12 technologies = 600 coordinates per snapshot, 3 per node installed.

That matters beyond the headline: the sparsity is structural and time-invariant — installed carries node and tech, never snapshot. A random Bernoulli mask sweeps the same densities while misrepresenting the shape, and the shape is what an engine can exploit.

The mask sweep — one model size, varying only the portfolio:

live	variables	wall: farkas	wall: linopy	wall	peak: farkas	peak: linopy	peak
100% (12 techs/node)	1.2M	1.22 s	0.39 s	3.10x	0.48 GB	0.63 GB	0.77x
50% (6)	600k	0.73 s	0.32 s	2.28x	0.35 GB	0.60 GB	0.58x
25% (3)	300k	0.44 s	0.28 s	1.56x	0.25 GB	0.45 GB	0.55x
8% (1)	100k	0.30 s	0.24 s	1.25x	0.20 GB	0.35 GB	0.59x

We cost what survives; the eager lane costs the product. Thinning the portfolio cuts our wall 4x and linopy's 1.6x. At nodal/3M the peak ratio is 0.38x, the best in the ladder.

The clock was measuring things only one arm does

A phase now counts only if both arms could pay it:

• linopy was being penalised. to_file defaults progress to m._xCounter > 10_000, so every rung above the smallest had been rendering tqdm progress bars that the farkas arm has no equivalent of — ~7% of the write at 10M variables, and stderr noise in a harness that parses stdout. Now progress=False.
• farkas was getting teardown for free. ex.close() releases the scratch database; a user pays for that, so it is timed and reported as its own phase.
• The parameter/dimension split re-parses the YAML purely because the runner decides which parquet file is which. Only farkas was paying it; it moved outside the clock.
• workdir_bytes is recorded, because a peak-RSS win should not quietly hide a multi-gigabyte temp file. 0.25 GB at 9.8M variables — small beside the LP file, but not zero.

Both corrections move against us. dispatch at 10M went 2.09x -> 2.96x, and almost none of that is the engine. The boundaries are now a table in bench/README.md instead of an accident of where the marks happened to sit.

Also

• --sizes all runs every rung a case has, since the density rungs only exist on masked cases.
• Parity gate passes on all three cases with bit-identical objectives.
• The 256MB OOM on transport still reproduces here (relevant to Constraint assembly OOMs instead of spilling when the group_sum fan-in is wide #145 and to the claim in perf: close the build-time gap, and measure the mask the ladder never measured #153 that it did not).

Overlap note: #153 is stacked on this branch and adds its own sparse.yaml with a fixed 20% Bernoulli mask. nodal supersedes it on realism and adds the density axis, but they will conflict in bench/cases.py — happy to reconcile in whichever lands second.