fix(constraints): CSRConstraint stores label columns; to_constraint on a CSR payload returns an unassigned CSRConstraint by FabianHofmann · Pull Request #929 · PyPSA/linopy

FabianHofmann · 2026-09-06T15:27:54Z

Closes #926. Supersedes #927.

(handwritten intent goes here)

Note

The following content was generated by AI (Claude Code), reviewed by @FabianHofmann.

Changes proposed in this Pull Request

CSRConstraint stored dense variable positions as CSR column indices, with a width fixed at freeze time. Any add_variables or remove_variables after a freeze=True constraint made model.matrices fail with inconsistent shapes. This PR rebases #927 onto the merged #870 and completes the remaining two points of #926.

Label columns (points 1, 3, 5 of #926, from #927)

CSRConstraint stores raw variable labels as columns (width model._xCounter). A shared helper maps labels to dense positions once at matrix assembly, for both the dense Constraint and the frozen path, and raises if a constraint references a removed variable.
The returned block keeps the stored data and indptr arrays by identity and preserves the int32 index dtype, so the persistent snapshot diff can still skip untouched frozen constraints.
netCDF: the writer tags _csr_columns: labels; the reader maps legacy position files back through the stored vlabels and raises on unknown tags.
CSRConstraint.indexes built an xarray Indexes without a variables mapping, which made ModelSnapshot.capture fail on any model with a frozen constraint.

Unassigned CSR constraints (points 2 and 4 of #926)

CSRConstraint.from_payload keeps the payload's label columns as they are and no longer needs the variable label index. It returns an unassigned constraint. CSRConstraint stores the flat grid positions of its active rows; labels are derived as cindex + position, and active_labels() raises while unassigned. Constraints.add rejects unassigned constraints.
LinearExpression.to_constraint on a CSR-backed expression returns that unassigned CSRConstraint directly. Constraint._from_pending, the lazy Constraint.data branch and extract_csr_pending from feat(v1): CSR-backed sparse groupby-sum - skew-independent build memory (6-9x on the #745 hub case) #870 are removed.
Model.add_constraints builds the anonymous constraint once via _constraint_from_lhs; for a CSRConstraint under freeze it only assigns labels, otherwise it materializes to the dense path as before. The sparse freeze path now also honours the scaling argument, which it previously ignored. It also runs the infinite-rhs, dimension-name and multiindex checks that the dense path runs, and it labels a group without terms like the dense path in both freeze modes (term-less rows are kept while unassigned and dropped on freeze, as from_mutable does).
CSRConstraint.data pads unused term slots in coeffs with NaN instead of 0, and constraints.FILL_VALUE["coeffs"] follows, matching the expressions fill value.
Return annotations of to_constraint, le/ge/eq and the comparison operators are widened to ConstraintBase.

Tests

Add/remove variables after freeze against the unfrozen model, netCDF round-trip incl. legacy layout, buffer identity and empty diff for an untouched frozen model, to_constraint on a CSR lhs returning an unassigned CSRConstraint that labels equivalently to the dense path, row scaling on the frozen path, errors on label access of an unassigned constraint, label parity of a group without terms for both freeze modes, and infinite-rhs rejection on both paths.

Verification

Check	Result
`ruff check` / `ruff format`	clean
`mypy linopy`	28 errors, all pre-existing in `solvers.py`
`pytest -k "not xpress" --ignore=test/remote`	6376 passed, 556 skipped

Checklist

AI-generated content is marked (see AGENTS.md).
Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

…s at matrix assembly (#926)

…signed CSRConstraint (#926) from_payload keeps the payload's label columns and no longer needs the label index; add_constraints only assigns labels (and honours scaling). Drops Constraint._from_pending, the lazy data branch and extract_csr_pending.

github-actions · 2026-09-06T15:30:59Z

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05998 (1.11)   │   15.03 (1.00) 
 (v1)        0.05423 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06649 (1.15)   │   12.04 (1.00) 
 (v1)        0.05802 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02638 (1.11)   │   15.20 (1.0) 
 (v1)        0.02386 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03965 (1.14)   │   44.93 (1.0) 
 (v1)        0.03465 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02906 (1.11)   │   11.51 (1.0) 
 (v1)        0.02615 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06806 (1.11)   │   24.34 (1.06) 
 (v1)        0.06156 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.07538 (1.10)   │   16.12 (1.00) 
 (v1)        0.06852 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.01596 (1.10)   │   752.18 (1.10) 
 (v1)         0.0145 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.01576 (1.10)   │   3.12 (1.33) 
 (v1)        0.01434 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04683 (1.27)   │   126.16 (1.44) 
 (v1)        0.03697 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04579 (1.25)   │   126.16 (1.44) 
 (v1)        0.03653 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04774 (1.29)   │   126.16 (1.44) 
 (v1)        0.03689 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0371 (1.05)   │    715.12 (1.0) 
 (v1)       0.03537 (1.0)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03533 (1.11)   │   4.54 (1.27) 
 (v1)        0.03172 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.2485 (1.04)   │   704.12 (1.09) 
 (v1)        0.2387 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.2646 (1.04)   │   18.34 (1.0) 
 (v1)         0.254 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.2613 (1.05)   │   9.54 (1.0) 
 (v1)         0.248 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05229 (1.14)   │   3.77 (1.12) 
 (v1)         0.0458 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05025 (1.09)   │   216.59 (1.10) 
 (v1)        0.04591 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.02672 (1.14)   │   938.49 (1.0) 
 (v1)        0.02342 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02688 (1.11)   │   9.66 (1.0) 
 (v1)        0.02415 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02719 (1.13)   │   5.32 (1.0) 
 (v1)        0.02399 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01571 (1.04)   │   1.35 (1.0) 
 (v1)        0.01514 (1.0)   │   1.35 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01531 (1.03)   │   1.35 (1.0) 
 (v1)        0.01491 (1.0)   │   1.35 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01529 (1.02)   │   1.35 (1.0) 
 (v1)        0.01497 (1.0)   │   1.35 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.1261 (1.06)   │   946.85 (1.06) 
 (v1)        0.1194 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1236 (1.05)   │   12.01 (1.00) 
 (v1)        0.1174 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03166 (1.06)   │   147.70 (1.06) 
 (v1)        0.02981 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03148 (1.07)   │   2.60 (1.09) 
 (v1)        0.02929 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02618 (1.09)   │   696.75 (1.03) 
 (v1)        0.02409 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06343 (1.03)   │   137.97 (1.0) 
 (v1)         0.0616 (1.0)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04368 (1.10)   │   69.22 (1.0) 
 (v1)        0.03963 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03071 (1.12)   │   402.33 (1.00) 
 (v1)        0.02739 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03035 (1.13)   │   3.19 (1.19) 
 (v1)        0.02691 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03373 (1.08)   │   29.00 (1.54) 
 (v1)        0.03112 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04227 (1.15)   │   37.95 (1.43) 
 (v1)        0.03667 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)    0.06292 (1.0)   │    410.93 (1.0) 
 (v1)       0.06505 (1.03)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.06658 (1.0)   │     9.94 (1.0) 
 (v1)         0.07 (1.05)   │   10.22 (1.03)

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

codspeed-hq · 2026-09-06T15:31:14Z

Merging this PR will regress 1 benchmark

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 14 improved benchmarks
❌ 1 regressed benchmark
✅ 166 untouched benchmarks
⏩ 181 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
❌	`test_to_lp[qp-n=1000]`	2 MB	2.6 MB	-23.34%
⚡	`test_to_lp[expression_arithmetic-n=10]`	1,338.6 KB	718.2 KB	+86.37%
⚡	`test_to_solver[highs-rolling-severity=100]`	245 MB	186.1 MB	+31.65%
⚡	`test_to_solver[highs-rolling-severity=50]`	183.9 MB	140.2 MB	+31.1%
⚡	`test_to_solver[gurobi-rolling-severity=100]`	245.3 MB	188.3 MB	+30.24%
⚡	`test_to_solver[highs-kvl_cycles-severity=0]`	309.9 MB	242.8 MB	+27.64%
⚡	`test_to_solver[highs-cumsum-severity=100]`	79.5 MB	63.4 MB	+25.52%
⚡	`test_to_solver[highs-kvl_cycles-severity=50]`	158.1 MB	128.9 MB	+22.62%
⚡	`test_to_lp[storage-n=250]`	37 MB	30.8 MB	+20.03%
⚡	`test_to_solver[gurobi-rolling-severity=50]`	184.1 MB	153.8 MB	+19.7%
⚡	`test_to_solver[highs-cumsum-severity=50]`	20.2 MB	17.1 MB	+18.58%
⚡	`test_to_solver[gurobi-kvl_cycles-severity=50]`	158.4 MB	134.2 MB	+18.04%
⚡	`test_to_solver[gurobi-kvl_cycles-severity=0]`	310.1 MB	268.9 MB	+15.32%
⚡	`test_to_solver[gurobi-cumsum-severity=50]`	20.5 MB	18 MB	+13.55%
⚡	`test_to_solver[gurobi-cumsum-severity=100]`	79.8 MB	70.6 MB	+12.96%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing fix/csr-label-columns-926 (3442e71) with master (b71e9a9)}

181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

… label the sparse freeze path like the dense one (#926) - store _active_positions instead of overloading _con_labels; active_labels() raises on an unassigned constraint and Constraints.add rejects it - from_payload keeps term-less rows so freeze=False matches dense labels; assign_labels drops them as from_mutable does - sparse freeze path runs the infinity, dim-name and multiindex checks - FILL_VALUE coeffs is NaN, matching expressions and CSRConstraint.data - missing-label guard also rejects the -1 sentinel

FabianHofmann added 3 commits September 6, 2026 15:13

fix(constraints): CSRConstraint.indexes builds a valid xarray Indexes

eb013e0

fix(constraints): CSRConstraint stores label columns; map to position…

c44fbc4

…s at matrix assembly (#926)

FabianHofmann mentioned this pull request Sep 6, 2026

fix(constraints): CSRConstraint stores label columns; frozen constraints survive later variable changes #927

Closed

This was referenced Sep 6, 2026

CSRConstraint: cache positional CSR so the snapshot diff keeps its identity fast path #933

Open

read_netcdf sorts containers alphabetically, permuting matrices.A after a round trip #934

Open

Navigation Menu

Uh oh!

fix(constraints): CSRConstraint stores label columns; to_constraint on a CSR payload returns an unassigned CSRConstraint - #929

fix(constraints): CSRConstraint stores label columns; to_constraint on a CSR payload returns an unassigned CSRConstraint#929
FabianHofmann wants to merge 4 commits into
masterfrom
fix/csr-label-columns-926

FabianHofmann commented Sep 6, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 6, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Sep 6, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

FabianHofmann commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this Pull Request

Checklist

Uh oh!

github-actions Bot commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build cost — v1 vs legacy

Uh oh!

codspeed-hq Bot commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 1 benchmark

Performance Changes

Footnotes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

FabianHofmann commented Sep 6, 2026 •

edited

Loading

github-actions Bot commented Sep 6, 2026 •

edited

Loading

codspeed-hq Bot commented Sep 6, 2026 •

edited

Loading