Skip to content

test(bench): recover the duckdb arm and the marginal-cost pass - #240

Merged
FBumann merged 1 commit into
polars-enginefrom
bench/recover-three-arm
Jul 28, 2026
Merged

test(bench): recover the duckdb arm and the marginal-cost pass#240
FBumann merged 1 commit into
polars-enginefrom
bench/recover-three-arm

Conversation

@FBumann

@FBumann FBumann commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Both were written against #231's branch and kept being committed there after that PR was squash-merged, so none of it reached polars-engine. --duckdb-root, --builds, the loop mode, FOREIGN_ARMS and the marginal-cost table were all absent. My mistake, and it is why #189's headline numbers cannot be refreshed yet.

The duckdb arm runs the engine this branch replaces from its own checkout, its own interpreter and its own bench/_run_case.py — nothing is reimplemented here — interleaved with the other two against one shared parquet cache, with the parity gate spanning all three.

The loop pass answers the question the timings cannot. The harness spawns one process per measurement, so every timing is a first build, and each lane does lazy first-call work: ~180 ms eager, ~21 ms duckdb, ~4 ms here. first is what a caller pays who builds one model and solves it; steady is what a rolling horizon pays for every model after. Both are real.

It measures build and hand-off rather than build alone — linopy defers ~82% of its work to whatever consumes the model, so a build-only loop compares our finished matrix against its placeholder. I published a wrong claim from exactly that flaw and it is fixed here.

Smoke-tested on the current base:

dispatch xs  farkas  first  10.4 ms  steady   5.8 ms  warm-up   4.5 ms
             linopy  first 196.0 ms  steady  14.3 ms  warm-up 181.7 ms
             duckdb  first  43.9 ms  steady  23.4 ms  warm-up  20.5 ms

bench/README.md keeps #233's structure — narrow runs go to /tmp so they cannot clobber the published file — with the three-engine invocation added to the publishing block.

No numbers are published here. The base has moved twice under them (#233, #234) and again with #238, and the linopy pin is now the v1-semantics build rather than 0.9.0. The ladder is re-run separately, against that linopy, and docs/benchmarks.md and #189 are updated from it.

417 passed / 1 xfailed, ruff and pyrefly clean.

🤖 Generated with Claude Code

Both were written against #231's branch and kept being committed there after
that PR was squash-merged, so none of it reached `polars-engine`:
`--duckdb-root`, `--builds`, the `loop` mode, `FOREIGN_ARMS` and the marginal
cost table were all absent. This lands them on the current base.

**The duckdb arm** runs the engine this branch replaces from its own checkout,
its own interpreter and its own `bench/_run_case.py`, interleaved with the
other two against one parquet cache, with the parity gate spanning all three.

**The loop pass** answers the second question the timings cannot: the harness
spawns one process per measurement, so every timing is a *first* build, and
each lane does lazy first-call work — ~180 ms eager, ~21 ms duckdb, ~4 ms
here. `first` is what a caller pays who builds one model and solves it;
`steady` is what a rolling horizon pays for every model after. Both are real,
so both are reported.

It measures build *and* hand-off, not build alone: linopy defers ~82% of its
work to whatever consumes the model, so a build-only loop compares our
finished matrix against its placeholder.

`bench/README.md` keeps #233's structure — narrow runs go to `/tmp` so they
cannot clobber the published file — with the three-engine invocation added to
the publishing block, which is what it is.

No numbers are published here: the base has moved under them twice (#233,
#234) and again with #238, and the linopy pin is now the v1-semantics build
rather than 0.9.0. The ladder is re-run separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 760233c1-8d67-408a-b24d-b00869baa7f6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch bench/recover-three-arm

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@FBumann
FBumann merged commit 45902ec into polars-engine Jul 28, 2026
3 checks passed
@FBumann
FBumann deleted the bench/recover-three-arm branch July 31, 2026 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant