fix(v1): sparse merge keeps absent cells absent instead of zero-filling (#928) by FabianHofmann · Pull Request #931 · PyPSA/linopy

FabianHofmann · 2026-09-06T16:02:23Z

Part of layer 0 in #756 (comment). Rather a small but needed fix.

Closes #928.

Note

The following content was generated by AI (Claude Code).

Root cause. CSRPayload._from_scatter zero-filled a NaN constant for every caller. That matches the dense group kernel in from_grouper (a NaN member counts as 0), but it is wrong in from_expression, which converts a dense summand during try_csr_merge: there a NaN const means the cell is absent and must propagate, as the dense v1 merge does via skipna=False and absorb_absence.

Fix.

_from_scatter takes a skipna flag: from_grouper passes True, from_expression passes False, so an absent cell keeps its NaN const in the payload.
materialize applies absorb_absence, so the dense expansion carries no terms in absent cells (the CSR rows still hold the other summand's terms).
CSRConstraint.from_payload needed no change: it moves the const into the rhs and already drops NaN-rhs rows.

Test. test_merge_keeps_absent_cell_absent merges a sparse group sum with a .where-masked dense one, asserts equality with the all-dense merge, and asserts the frozen CSRConstraint drops the masked rows.

Repro from the issue, after the fix

sparse=False: const=[ 0. nan] rows=1
sparse=True:  const=[ 0. nan] rows=1

…ng (#928) CSRPayload.from_expression no longer zero-fills a NaN const when converting a dense summand; materialize applies absorb_absence so absent cells carry no terms, matching the dense v1 merge path.

codspeed-hq · 2026-09-06T16:05:41Z

Merging this PR will improve performance by 82.45%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 180 untouched benchmarks
⏩ 181 skipped benchmarks¹

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	`test_to_lp[expression_arithmetic-n=10]`	1,338.6 KB	733.7 KB	+82.45%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing layer0-928 (7abdd41) with master (b71e9a9)}

181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

github-actions · 2026-09-06T16:05:57Z

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09186 (1.13)   │   15.03 (1.00) 
 (v1)        0.08136 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09542 (1.09)   │   12.04 (1.00) 
 (v1)        0.08772 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03907 (1.10)   │   15.20 (1.0) 
 (v1)        0.03548 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05632 (1.10)   │   44.93 (1.0) 
 (v1)        0.05119 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04256 (1.08)   │   11.51 (1.0) 
 (v1)        0.03926 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1055 (1.12)   │   24.34 (1.06) 
 (v1)       0.09464 (1.0)   │    23.05 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1111 (1.06)   │   16.12 (1.00) 
 (v1)        0.1044 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02412 (1.08)   │   752.18 (1.10) 
 (v1)        0.02233 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02355 (1.07)   │   3.12 (1.33) 
 (v1)        0.02203 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06565 (1.25)   │   126.16 (1.44) 
 (v1)        0.05244 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06461 (1.23)   │   126.16 (1.44) 
 (v1)        0.05256 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06486 (1.21)   │   126.16 (1.44) 
 (v1)        0.05341 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0555 (1.02)   │    715.14 (1.0) 
 (v1)       0.05459 (1.0)   │   787.74 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05375 (1.09)   │   4.55 (1.27) 
 (v1)         0.0491 (1.0)   │    3.59 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                time (s)          peak (KiB) 
 name               mean   │            mean 
─────────────────────────────────────────────
 (legacy)   0.375 (1.05)   │   704.12 (1.09) 
 (v1)        0.357 (1.0)   │    647.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.3959 (1.05)   │   18.34 (1.0) 
 (v1)        0.3769 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.3941 (1.06)   │   9.54 (1.0) 
 (v1)        0.3706 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.07599 (1.08)   │   3.77 (1.12) 
 (v1)        0.07021 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0762 (1.08)   │   216.59 (1.10) 
 (v1)       0.07047 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03897 (1.09)   │   938.49 (1.0) 
 (v1)        0.03563 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.04009 (1.09)   │   9.66 (1.0) 
 (v1)        0.03671 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03993 (1.11)   │   5.32 (1.0) 
 (v1)        0.03601 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02123 (1.02)   │   1.47 (1.0) 
 (v1)        0.02083 (1.0)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.0212 (1.02)   │   1.47 (1.0) 
 (v1)       0.02078 (1.0)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02106 (1.01)   │   1.47 (1.0) 
 (v1)        0.02084 (1.0)   │   1.47 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.1911 (1.06)   │   946.85 (1.06) 
 (v1)        0.1796 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)        peak (KiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.1861 (1.04)   │   12.01 (1.0) 
 (v1)        0.1782 (1.0)   │   12.01 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04896 (1.09)   │   147.70 (1.06) 
 (v1)         0.0449 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04831 (1.09)   │   2.60 (1.09) 
 (v1)        0.04437 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03966 (1.09)   │   696.75 (1.03) 
 (v1)        0.03624 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09018 (1.06)   │   137.97 (1.0) 
 (v1)        0.08536 (1.0)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.06116 (1.09)   │   69.22 (1.0) 
 (v1)          0.056 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04594 (1.11)   │   402.33 (1.00) 
 (v1)        0.04124 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04582 (1.12)   │   3.19 (1.19) 
 (v1)          0.041 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04966 (1.06)   │   29.00 (1.54) 
 (v1)        0.04668 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06022 (1.10)   │   37.95 (1.43) 
 (v1)        0.05453 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)    0.09421 (1.0)   │    410.93 (1.0) 
 (v1)       0.09614 (1.02)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)    0.1017 (1.0)   │     9.94 (1.0) 
 (v1)       0.1043 (1.02)   │   10.22 (1.03)

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

fix(v1): sparse merge keeps absent cells absent instead of zero-filli…

7abdd41

…ng (#928) CSRPayload.from_expression no longer zero-fills a NaN const when converting a dense summand; materialize applies absorb_absence so absent cells carry no terms, matching the dense v1 merge path.

FabianHofmann merged commit 96dd488 into master Sep 6, 2026
25 checks passed

FabianHofmann deleted the layer0-928 branch September 6, 2026 16:27

Navigation Menu

Uh oh!

fix(v1): sparse merge keeps absent cells absent instead of zero-filling (#928) - #931

fix(v1): sparse merge keeps absent cells absent instead of zero-filling (#928)#931
FabianHofmann merged 1 commit into
masterfrom
layer0-928

FabianHofmann commented Sep 6, 2026

Uh oh!

codspeed-hq Bot commented Sep 6, 2026

Uh oh!

github-actions Bot commented Sep 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

FabianHofmann commented Sep 6, 2026

Uh oh!

codspeed-hq Bot commented Sep 6, 2026

Merging this PR will improve performance by 82.45%

Performance Changes

Footnotes

Uh oh!

github-actions Bot commented Sep 6, 2026

Build cost — v1 vs legacy

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant