feat!: adopt linopy v1 semantics — absence propagates and drops the row by FBumann · Pull Request #234 · FBumann/farkas

FBumann · 2026-07-28T09:06:04Z

Stacked on #189 (base polars-engine). Closes #8 and #219.

Breaking: absence propagates and drops the row. A term whose variable is
masked out no longer contributes zero — it makes the row absent.

x - rel_max * size <= 0     # size masked out on an unsized component

before:  c1: +x1 <= 0       # the flow is pinned to zero. feasible, plausible, no error
after:   (no row)

That silent pin is goal 1 of linopy's v1 convention ("no silent wrong answers")
and the whole of PyPSA/linopy#712 — and it was reachable here, not hypothetical.

What does not propagate

Getting these wrong would break real models, so they are rules, not omissions:

a reduction skips absent slots (§13), so sum(x, over=d) stays defined
when only part of d exists — otherwise one masked component deletes a
system-wide accounting row
a parameter covering some coordinates is sparse encoding, not absence:
missing rows mean a zero coefficient (SPEC §8), which is what lets a
coefficient table hold live entries only

Absence is a property of variables. SPEC §6 now says so.

The escape hatch, and why it is not optional

Dropping the row is now the only reading available, so a model wanting the other
one needs to name those coordinates. A bare parameter name in a where asks
"does this have a value here"; a bare variable name now asks "does this
exist here" — same grammar, and resolution already had the branch:

- expression: x - rel_max * size <= 0
  where: "size"
- expression: x <= 0
  where: "NOT size"

Without it the only spelling is a parameter mirroring the variable's own mask —
two sources for one fact, and when they drift the failure is a row quietly
pinned to zero rather than an error.

How the engine separates the two

Once coeff × var are multiplied the frame cannot say which side removed a row,
so this could not be an anti-join over the term stream. TermFragment.presence
carries the variable's own coordinates alongside, rewritten by the same shape
operators.

Populated only for a variable whose declaration has a where — decided off
the plan before data is read. That guard is load-bearing: a presence frame is
data, so it costs _label_frame both of its arithmetic paths, and those are
recently merged work (#152, #178). Equations that do not turn on the difference
keep them.

The oracle is v1, and it raises rather than skips

A differential test is an oracle only if it compares against the convention we
implement. Legacy fills absent slots with zero — it agrees with the old
reading — so measuring against it would pin this package to behaviour upstream
classifies as a bug. tests/oracle.py therefore raises on a linopy without
options["semantics"]; a skip would go green having stopped checking the lanes
on exactly the cases that changed.

No release carries the option, so [tool.uv.sources] pins PyPSA/linopy#717 by
branch, not by rev — this tracks active upstream work and a stale rev would
measure us against a spec that has moved.

Two findings worth keeping

§8 cannot fire in this language. Verified: four orderings of
a + factor + b with a sparse constant and a masked variable give
byte-identical rows. The master coordinate (SPEC §8) removes the ambiguity §8
exists to catch, one layer earlier; PyPSA/linopy#711's break needs chained
left-joins reindexing one operand onto another, and we never chain.

§5 fires on sparse parameters, which no test covered — the 414-green suite
had no sparse float parameter. farkas/linopy/semantics.py answers absence per
position: zero in a coefficient, error in bounds:, false in a where. A
single fill in the loader is wrong for two of the three. (The loader had already
made exactly this call for booleans, one line above.)

Bug fixed on the way

A variable's own where naming itself was a KeyError: 'p' out of linopy —
nothing in this package's voice. Found by the parity sweep, which masks
variables.p.where and so asked p whether p exists. Now a load error: that
mask is what decides where the variable exists, so it cannot depend on it.

Verification

417 passed, 1 xfailed, on the pinned environment. uv sync → suite → ruff →
pyrefly all clean.

Known limits

defined(v) refuses a variable whose foreach exceeds the frame — masking a
[snapshot] constraint by a [snapshot, generator] variable would silently
widen the mask, and saying which reduction is meant needs ROADMAP Track 1 item 6
(all(x, over=d)). For the same reason the predicate has no home in the
dispatch-model parity sweep, so COVERED_ELSEWHERE maps it to the test that does
exercise it rather than weakening the coverage guard.

test_a_constraint_row_left_with_no_variables stays xfailed and is not this
issue: raw linopy builds a term-less row under both conventions (labels=[0,1],
vars=[-1]), so that divergence is in our own eager lane and wants its own
diagnosis.

…the answer The oracle lane runs clean under `linopy.options["semantics"] = "v1"`, which issue #8 has been parked on since the tracked PR (PyPSA/linopy#591) was closed unmerged. The live work is PyPSA/linopy#717; its spec is `doc/design/ convention.rst` on `feat/arithmetic-convention`. Two things the convention changes for us, and one it turns out it cannot: **§8 never fires.** Its "shared dimensions must carry identical labels" exists because linopy operands are arbitrary xarray objects with independent indexes, so a mismatch is ambiguous between "different data" and "subset". We resolve a master coordinate per dimension before any data binds (SPEC §8) and reindex every operand to it, so a superset already raises, a subset is filled and a reorder is fixed by label. Measured: four orderings of `a + factor + b` with a sparse factor and a masked variable produce byte-identical rows, so the associativity break v1 §8 exists to stop (PyPSA/linopy#711 — chained left-joins dropping coordinates) has no mechanism here. **§5 does fire, on our central idiom.** `load_parameters` reindexes to the master coordinates, so a parameter covering a subset of its dims arrives at linopy as NaN — and v1 refuses a NaN in a user-supplied constant, because from inside linopy a deliberate absence and a data error are indistinguishable. That is every sparse parameter, which is what SPEC §8 means by "sparse data gives sparse variables" and what a generated binder emits by the tableful. The 414 tests passing beforehand were not evidence otherwise; the suite had no sparse float parameter. It has one now, and it fails without this change. **So the answers are given per position, not once in the loader.** The same missing row means three different things: zero in a coefficient, an error in `bounds:` (unbounded is not bounded-at-zero), false in a `where` (SPEC §6's bare name is "non-null and finite"). A single fill in the loader would have to pick one and be wrong for the other two. `linopy/semantics.py` is where those answers live, mirroring linopy's own module of that name and for its reason: the evaluator stays about evaluating, and a later change to the convention is a single-file diff. All three functions are correct under the legacy convention too — they resolve absence to the value legacy reached implicitly — so none is conditional on the option. What v1 changed is that staying silent stopped being available. Verified on three configurations: released linopy 0.9.0 (415 passed), the v1 branch under `semantics="v1"` (415 passed), and the same branch under `semantics="legacy"` (413 passed, 2 failing only on `LinopySemanticsWarning` promoted to an error by our own filterwarnings). Refs #8

coderabbitai · 2026-07-28T09:06:14Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 9476cdb0-e28b-4af4-9c69-cdb0b77f0307

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/linopy-v1-semantics

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…part Prerequisite for adopting v1's absence semantics, and inert on its own — no behaviour changes, 415 tests unmoved. A term fragment can lose rows at a coordinate for two unrelated reasons, and a constraint row has to react to exactly one of them. A **masked variable** is genuinely absent there. A **sparse parameter** is a compressed dense array whose missing rows mean a zero coefficient — SPEC §8's "sparse data gives sparse variables", and what a generated binder emits by the tableful. Once the two are multiplied into one frame the distinction is gone, which is why "drop the row where a variable is absent" could not be written as an anti-join against the term stream. So the variable's own coordinates ride alongside as `TermFragment.presence`, rewritten by the same shape operators that rewrite the term. The propagation rule follows from the convention rather than from convenience: variable leaf the variable's frame parameter, constant none — sparsity is an encoding, not absence a * b, a / b, -a the variable side's, unchanged: a sparse coefficient zeroes a term, it does not unmake the variable under it sum, group_sum cleared — §13 has reductions *skip* absent slots rather than propagate them, which is what keeps a cross-module accounting equation summing over a partly-masked dim roll, shift still to do: the coordinate map has to be applied to presence too, and shift's vacated edge unioned back in, since SPEC §7 declares it contributes zero Refs #8

Adopts linopy's v1 reading of absence as the language's own, and makes v1 the oracle rather than a mode we happen to survive. **What changes.** A term whose variable is masked out no longer contributes zero — it makes the row absent, so `x + y >= 10` is *no constraint* where `y` is masked rather than `x >= 10`. The old reading is how x - rel_max * size <= 0 silently became `x <= 0` on an unsized component: feasible model, plausible answer, no error. That is goal 1 of the v1 convention ("no silent wrong answers") and the whole of PyPSA/linopy#712, and it was reachable here. **Two things deliberately do not propagate.** A *reduction* skips absent slots (§13), so `sum(x, over=d)` stays defined when only some of `d` exists — without that, one masked component would delete a system-wide accounting row. A *parameter* covering only some coordinates is sparse encoding, not absence: its missing rows mean a zero coefficient (SPEC §8), which is what lets a coefficient table hold live entries only. Absence is a property of variables. **How the engine tells them apart.** `TermFragment.presence` carries the variable's own coordinates beside the term stream, because once `coeff x var` are multiplied the frame cannot say which side removed a row. It is set only for a variable whose declaration has a `where` — decided off the plan, before data — so an unmasked variable never imposes the cost, and `_label_frame` keeps both of its arithmetic paths (#152, #178) for every equation that does not turn on the difference. **The oracle is v1, and it raises rather than skips.** A skip would be the worst outcome available: the suite would go green having stopped comparing the lanes on exactly the cases the convention changed. No release carries the option yet, so `[tool.uv.sources]` pins PyPSA/linopy#717 by branch — by branch and not by rev on purpose, since a stale rev would measure us against a spec that has moved. Not included, and it is the follow-up this needs: `defined(v)` (#219). Dropping the row is now the only reading available, and the way to ask for the other one is complementary `where` clauses over a variable's existence — which is not yet sayable. Until it lands, a model wanting "keep the row, treat the term as zero" has to carry a parameter mirroring the variable's mask. `test_a_constraint_row_left_with_no_variables` stays xfailed and is *not* this: raw linopy builds a term-less row under both conventions (`labels=[0,1]`, `vars=[-1]`), so that divergence lives in our own eager lane and wants its own diagnosis. Refs #8, #219

The escape hatch the previous commit made necessary. Absence now takes the row with it, so a model wanting the other reading — keep the row, treat the term as zero — needs a way to name those coordinates. Without it the only spelling is a parameter mirroring the variable's own mask: two sources for one fact, and when they drift the failure is not an error but a row quietly pinned to zero. Surface is the one the language already implies rather than a new one. A bare *parameter* name in a where asks "does this have a value here"; a bare *variable* name asks "does this exist here". Same grammar, same shape, and resolution already had a branch for the case — it just raised. - expression: x - rel_max * size <= 0 where: "size" - expression: x <= 0 where: "NOT size" Pointwise on both lanes: a semi-join against the variable's frame relationally, `labels != -1` eagerly. `_predicate_dims` reads it through the variable's `foreach` exactly as a parameter is read through its dims, so `_free_prefix` keeps its arithmetic path for the leading dims a mask cannot see. **A self-reference is now a load error**, found by the parity sweep, which masks `variables.p.where` and so asked `p` whether `p` exists. It died in linopy with `KeyError: 'p'` — nothing in this package's voice. A variable's own where cannot ask whether it exists, because that mask is what decides it. Two limits, both deliberate and both visible in the tests. The dim rule refuses a variable whose foreach exceeds the frame — masking `balance` over `[snapshot]` by `p` over `[snapshot, generator]` would silently widen the mask, and saying which reduction is meant needs ROADMAP Track 1 item 6 (`all(x, over=d)`). And the predicate has no home in the dispatch-model parity sweep for that same reason, so `COVERED_ELSEWHERE` maps it to the test that does exercise it rather than letting the coverage guard be quietly weakened. Closes #219. Refs #8

Both were written against #231's branch and kept being committed there after that PR was squash-merged, so none of it reached `polars-engine`: `--duckdb-root`, `--builds`, the `loop` mode, `FOREIGN_ARMS` and the marginal cost table were all absent. This lands them on the current base. **The duckdb arm** runs the engine this branch replaces from its own checkout, its own interpreter and its own `bench/_run_case.py`, interleaved with the other two against one parquet cache, with the parity gate spanning all three. **The loop pass** answers the second question the timings cannot: the harness spawns one process per measurement, so every timing is a *first* build, and each lane does lazy first-call work — ~180 ms eager, ~21 ms duckdb, ~4 ms here. `first` is what a caller pays who builds one model and solves it; `steady` is what a rolling horizon pays for every model after. Both are real, so both are reported. It measures build *and* hand-off, not build alone: linopy defers ~82% of its work to whatever consumes the model, so a build-only loop compares our finished matrix against its placeholder. `bench/README.md` keeps #233's structure — narrow runs go to `/tmp` so they cannot clobber the published file — with the three-engine invocation added to the publishing block, which is what it is. No numbers are published here: the base has moved under them twice (#233, #234) and again with #238, and the linopy pin is now the v1-semantics build rather than 0.9.0. The ladder is re-run separately. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Every number in this file predated #233, #234 and #238, and was measured against linopy 0.9.0 rather than the v1-semantics build this branch now pins. Re-taken in one run: five cases, two sinks, three arms, best of three, plus the marginal-cost pass. 408 timings and 72 loop records, no failures. **Ahead on both axes on every case through the hand-off**, which is the sink most callers use: wall 0.35x, 0.32x, 0.25x, 0.45x, 0.86x and peak 0.95x, 0.84x, 0.32x, 0.76x, 0.88x. `profiled` was the one exception until #238 stopped the duplicate-coordinate check grouping 12M rows to answer a yes/no question; it is a win now like the rest. The LP file is the weaker route and stays that way — 0.71x to 1.35x on wall, and `transport` 1.61x on peak — because most of an LP write is float-to-text, work neither lane avoids. Against duckdb: 2.2-5.2x faster on every case and both sinks, and duckdb 1.2-2.9x lighter. duckdb is slower than the eager lane everywhere here. The parity gate agrees across all three arms at 0.0e+00 on every shared case, which is also what proves this branch's v1 semantics (#234) and `main`'s (#239) build the same model rather than merely claiming to. Provenance now names a commit per arm rather than a version, since two of the three arms have no meaningful version string — the duckdb arm is a checkout and the polars arm is an editable install that reports its sync point. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

FBumann added 2 commits July 28, 2026 11:12

FBumann changed the title ~~feat(compat): answer linopy's v1 convention where the position knows the answer~~ feat!: adopt linopy v1 semantics — absence propagates and drops the row Jul 28, 2026

FBumann mentioned this pull request Jul 28, 2026

feat(compat): answer linopy v1 where the position knows the answer #236

Merged

FBumann marked this pull request as ready for review July 28, 2026 10:55

FBumann merged commit 4ff049d into polars-engine Jul 28, 2026
2 of 3 checks passed

This was referenced Jul 28, 2026

perf(engine): ask whether a coordinate repeats before asking which one #238

Merged

feat!: absence propagates and drops the row, plus defined(v) #239

Merged

test(bench): recover the duckdb arm and the marginal-cost pass #240

Merged

This was referenced Jul 28, 2026

feat!: rebuild the relational engine on polars #189

Merged

docs(bench): re-run all three engines on the current base #243

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: adopt linopy v1 semantics — absence propagates and drops the row - #234

feat!: adopt linopy v1 semantics — absence propagates and drops the row#234
FBumann merged 4 commits into
polars-enginefrom
feat/linopy-v1-semantics

FBumann commented Jul 28, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 28, 2026 •

edited

Loading

Review skipped

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

FBumann commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does not propagate

The escape hatch, and why it is not optional

How the engine separates the two

The oracle is v1, and it raises rather than skips

Two findings worth keeping

Bug fixed on the way

Verification

Known limits

Uh oh!

coderabbitai Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

FBumann commented Jul 28, 2026 •

edited

Loading

coderabbitai Bot commented Jul 28, 2026 •

edited

Loading