Build models from math-spec programs, fold named expressions, round trip through netcdf - #922
Draft
FabianHofmann wants to merge 9 commits into
Draft
Build models from math-spec programs, fold named expressions, round trip through netcdf#922FabianHofmann wants to merge 9 commits into
FabianHofmann wants to merge 9 commits into
Conversation
Build cost — v1 vs legacyv1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).
Full table (time + peak, mean)📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run. Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped. |
4 tasks
FabianHofmann
changed the title
…ssions Port lpspec's linopy lane onto the binder: builder, where, operators, coverage and curves, wired to Bound and SpecDataError. Add Model.add_spec, Model.from_spec and the model.spec accessor with expressions and evaluate.
…en windows Coverage and the retain closure now descend into a Power's operands; evaluate() refuses sources labelled unlike the model; an all-null window width is a window of nothing; cases fold through the aligned combine.
Persist the spec text, the master coordinates and the lookups alongside the model, re-lowering the program from the text on read; math-spec is imported only for a file that carries a spec. Lookups and arrays of labels are stored as codes into a category table, so partial maps keep their holes and dtypes.
Write the in-memory dtype of every parameter and cast it back on read, and stamp the master coordinates onto every container, so no engine leaves a model disagreeing with itself. assert_model_equal now compares dataset dtypes, and synthetic_sources moves to linopy/spec/testing.py for both users.
FabianHofmann
force-pushed
the
spec-builder
branch
from
September 3, 2026 18:39
7fbfd88 to
e0a29c9
Compare
A missing parameter row was read as a silent zero when it stood as a coefficient, while a bound, constant side or divisor already refused it. Refuse it as a coefficient too, so every position behaves alike and a hole is never filled without the modeller saying so: mask the coordinate out with a where, or fill the value into the data.
A runnable, nbconvert-clean walkthrough of the spec feature: the dispatch program, binding data, folding named expressions, retain and evaluate, the uniform absence rule, lookups and grouped sums, temporal shift, and the netCDF round trip.
m.spec.expressions[name] returns a NamedExpression bundling .node (the lowered formula), .expression (the unsolved linopy expression) and .solution (the fold over the model's solution). evaluate() returns the same object. Add ModelSpec.to_latex/to_markdown/to_typst for whole-model typesetting, rendered as Markdown in a notebook.
building-models-from-specs.ipynb imports math_spec, which the docs CI environment does not install, so the notebook job failed on import. Skip it like the other special-setup notebooks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Steps 3 and 4 of the plan in #919, stacked on #921: build an empty linopy model from a math-spec YAML plus data, read named expressions back numerically after solve, and carry the spec through netcdf and
Model.copy().Note
The following content was generated by AI.
Changes proposed in this Pull Request
lpspec's linopy lane (fluxopt/lpspec#1518) ported into
linopy/spec/on top of the binder from #921, typed, wired toBoundandSpecDataError.Model.add_spec(spec, sources, retain="report")builds variables, SOS, constraints and objective into an empty model.specis a path, YAML text, dict ormath_spec.Spec, never aProgram.Model.from_spec(spec, sources, retain=..., **model_kwargs)is sugar overModel(**model_kwargs).add_spec(...). Spec-built models are v1;add_specraises under legacy semantics or on a non-empty model.model.spec(ModelSpec) holds theProgram, the spec text and reads coordinates, lookups and retained parameters back frommodel.parameters.model.spec.expressions[name]folds a named expression numerically after solve, substitutingVariable.solutionand retained parameters.model.spec.evaluate(name, sources)rebinds from fresh sources forretain="none", refusing labels the model was not built on.builder(declarations, fold),operators(sum_over,grouped_sum,at,shift,sum_back),where(masks, bound lookups),coverage(bounds, constant sides and divisors must cover their rows),curves(piecewise derivation and validation on xarray),terms(absence: a missing row is 0 as a coefficient, refused as bound or divisor),context,nodes(node walk that also descends intoPower, which math-spec alpha.73 skips),accessor,netcdf(encode/decode for persistence),testing(synthetic_sourcesfor tests and benchmarks).Dualnode (waits for feat(language): an expression the math never reads may be nonlinear, a reported quantity energy-models/math-spec#287), lpspec'sreads_off_the_solutiongrade predicate (linopy always folds), the polars loader (replaced by the binder). Variable divisors and degree-2 named expressions are refused by math-spec itself at lowering.SpecDataError); a hole an operator leaves beside a term drops the row; a constraint with its term on the right is swapped with a flipped sense.import linopystill never imports math_spec; the model methods import it lazily.Persistence (step 4):
to_netcdfwrites the spec YAML text as the top-level attribute_linopy_spec, next to_linopy_version, following the relaxed-registry and piecewise-formulation precedent.read_netcdfre-lowers theProgramfrom that text; a file without a spec loads without math-spec installed.spec-carries whatmodel.parameterscannot round-trip on its own: master coordinates ascoords__<dim>data variables (an index coordinate no retained parameter reaches would otherwise be dropped on read), and lookups plus object-dtype parameters aspandas.factorizeoutput,codes__<name>(int32, -1 for a hole) pluscats__<name>. Every parameter records its in-memory dtype in a variable attribute and is cast back on read. No netcdf type carries an object array of strings with NaN: netCDF4 returns<Uand turns a hole into an empty string, scipy narrows int64 to int32. Codes plus categories round-trip str->str, str->int, int->str and int->int lookups, full and partial, exactly on both engines. This resolves the two lookup items left open in feat(spec): xarray data binder for math-spec programs #921.Model.copy()carries the spec; the accessor is rebound to the copy. On read, the master coordinates are stamped onto every container, so one dimension has one dtype even after the scipy engine narrows int64 to int32.assert_model_equalcompares the spec text and the parameter datasets including dtypes.test/test_spec_io.py: round trips forretain="report"|"all"|"none"on both engines, a solved model whose named expression is identical before and after,evaluateon a restoredretain="none"model, exact-dtype assertions per lookup kind, copy semantics, and math-spec'sexamples/pypsa.yamlwith synthetic data whenMATH_SPEC_EXAMPLESpoints at a math-spec checkout.benchmarks/models/spec_pypsa.py:from_speconexamples/pypsa.yamlplus the two netcdf phases; skips without math-spec orMATH_SPEC_EXAMPLES. Running it in CI needs math-spec installed and the examples directory available in the codspeed workflow, which waits for a math-spec PyPI release.Found upstream: math-spec alpha.73
program.children()does not descend intoPower, soparameters_of/walkmiss parameters under**. Worked around inlinopy/spec/nodes.py.Verification
Checklist
AGENTS.md).doc.doc/release_notes.rstof the upcoming release is included.