test(bench): carry over the profiled case and the sink axis, and re-measure on both

[FBumann]

Carries the two ideas from #205 onto polars-engine, and re-measures everything on them. Based on polars-engine after #211.

#205 does not cherry-pick. It also carries --memory-limits, --chunk-rows and workdir_bytes — the duckdb budget apparatus #189 deleted — so taking the commit would have reintroduced it. Both ideas are ported by hand instead.

The headline: the sink decides the answer

transport's peak is 1.62x through the LP writer and 0.78x through the hand-off. Same build, same rung. This file has called that row "the one open number at scale" — on the sink most callers actually use, we are ahead of linopy on it.

l rung, wall / peak:

	lp	highs
dispatch	0.72x / 1.03x	0.36x / 0.95x
nodal	0.63x / 0.83x	0.32x / 0.80x
sector	0.37x / 0.32x	0.25x / 0.32x
transport	0.86x / 1.62x	0.46x / 0.78x
profiled	1.74x / 1.14x	1.13x / 0.94x

Ahead on both axes on four of five cases through the hand-off. The LP path spends its peak turning doubles into text; HiGHS's own dense model is resident in both arms and dwarfs the difference between the lanes that filled it. The "what this says" section now reads per sink, because the two disagree by more than the noise between them.

What is in it

profiled — the one case whose input is the same order as its model: availability dense over (snapshot, node, tech), a 12M-row table against a 12M coordinate product at l. It is the shape the eager lane handles best, and it is the case we lose (1.74x LP wall). That is why it is here — a ladder holding only shapes that suit one engine proves nothing.

--sinks lp highs, both by default, and build_highs split out of solve_direct to give it a seam that stops before run(). This is what makes the build-and-hand-off section reproducible: it was a hand-made table with no script behind it, carrying a note from me saying its peak column was stale and could not be re-run without one. That note and that table are both gone.

Two things it found

A real bug. profiled writes its dim columns dictionary-encoded — what pandas does for any Categorical, and what any writer does for a 12M-row table of repeated labels. polars will not join Categorical against String, and dim frames are plain, so binding such a source died with a schema error from inside a join. duckdb coerced silently, so this arrived with #189 rather than being found by it. Source dim columns are now cast at bind time; the test fails without it.

The density sweep no longer refuses its own claim. Peak improves at every rung now (0.87x → 0.74x) where linopy used to win the sparsest one. Not because the prediction changed but because a mask got cheaper for us — #212 made masked labels arithmetic, and the sweep had been measuring our own cost of being sparse.

Provenance

294 records, five cases, two sinks, both arms, best of three, one invocation, parity gate green on all five at 0.0e+00 relative. Machine quiet. 401 passed / 1 xfailed, ruff and pyrefly clean.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 12125b36-20c8-4f07-a555-6ea0d69e1df9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch bench/carry-over-profiled-and-sinks

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[FBumann]

Rebased onto polars-engine (picking up #225), and CI fixed.

Three things were wrong:

• ruff format --check — I ran ruff check but not ruff format on the final state, and ruff check --fix had reintroduced two files' worth of drift after my last format pass. Fixed.
• PR title type. bench: is not in the allowed set (feat, fix, perf, refactor, docs, chore, test, ci, build, style, revert). Retitled to test(bench):.
• The branch was behind.

While fixing the title I split the commits rather than just renaming them, because the original lumped a user-facing bug fix under a bench: subject — it would have landed in the changelog as harness work, or not at all:

fix: a dictionary-encoded source column binds like a plain one
refactor(sinks): split build_highs out of solve_direct
test(bench): a case where I/O is not noise, and a sink axis to measure it on
docs(bench): re-measure on both sinks, and let the sink decide the answer

Each one is independently green (404 passed / 1 xfailed at every commit), so the history bisects.