feat(language): an expression the math never reads may be nonlinear, a reported quantity

[FabianHofmann]

Prompt: implement dev/postsolve-expressions/plan-unify-postsolve.md

Prompt: The flag should say if it's part of the math problem or only a report quantity

Note

The following content was generated by AI.

One expressions: section, no postsolve:. An entry is reported when the math never reads it — nothing in the objective or a constraint inlines its body, directly or through another entry, a macro, or a piecewise link. A reported entry may be nonlinear and prints in a flag-gated "Reported quantities" section as symbol = body. Folds in the dual(constraint) builtin from #290, which can now close.

Usage, not shape. Whether an entry is reported is decided by use, not by the shape of its body. expansion.read_by_the_math(schema) expands the objective and every constraint and collects each named entry inlined on the way — through another entry, a macro's call-by-value substitution, or a piecewise link's emitted constraint. Lowering stores the answer on each ExpressionDeclaration.in_math, and the typesetter asks the same function for its "Reported quantities" section, so the program and the printed page cannot disagree about which entries the math reads. An entry is degree-checked only where the math reads its inlined body — every constraint, the objective, each where, bound and piecewise link — at that position's ceiling, so a nonlinear entry named by a constraint is refused there, not banned at declaration.

"Reported", not "post-solve". "Reported" is what is true of lcoe, dual(...), and a data-only case like (1 + rate) ** period alike: a quantity read back, never fed to a solve. A reported quantity need not need a solve, so "post-solve" named a stage it does not have.

The dual. dual(constraint) is a constraint's shadow price, a number only a solve produces. Resolution types it to a single DualNode leaf, and placement is one guard — degree.calls_dual, run on the resolved tree at every math-building position, refusing a dual there with a message that names the rewrite. The expressions: declaration is the one position that does not run it — that is where a dual belongs. Its only exposed form is the resolved program.Dual; the parsed node is exported nowhere.

Verified: pixi run ci green at f8eca1c — lint, full test suite, mkdocs build --strict, 28 TeX documents compiled. Schema unchanged (neither the reported flag nor dual adds a key); schema and golden regenerated with no drift.

Why the flag is usage, not a shape grade

An earlier revision decided "reported" by grading the body's shape — a variable divisor or exponent, a product above degree 2, or a dual() call — at the math's degree-2 ceiling. The grade answered "could the math read this body", which is not the section's claim: reported.md's invariant is that the math never reads a reported entry. So an affine entry nothing references graded math-grade and printed nowhere, though nothing reads it. in_math answers the question the section actually asks, and the predicate is one a consumer can reach (a flag on the lowered declaration) rather than one taking a private parser node no second reader could call. The consequence a reader sees: an affine entry nothing references now prints under Reported quantities, and a symbol table may rename it.

Rebase — onto the internalized parsers

The work sits on main's parser internalization (#342 + #351): the dual edits _expression_parser.py (private), resolution is main's _Resolver, the dim rule is a branch in _dims_call dispatch, and the typeset dual leaf and section builder use main's walk API. dual resolving to a DualNode leaf lets dimensions, lowering, the typesetter, carries_variable and the template-name check each drop a "a bare constraint reference cannot reach here" guard and the dual special case in call dispatch, keeping one arm.

Tests and departures

• Where the math reads decides refusal. test_a_nonlinear_entry_is_refused_where_the_math_reads_it (constraint, objective), a piecewise-link case at ceiling=1, and a bound/where naming no expression. TestDual covers the placement refusals, including a dual smuggled through a macro or an entry, now on the resolved tree.
• in_math across reference shapes in test_lowering.py: a constraint, the objective, through another entry, through a macro, unreferenced, and referenced only by a reported entry; plus the chain (intermediate and leaf both in the math), a macro formal sharing an entry's name (the argument decides), and an entry reading a dual (reported, a Dual leaf). calls_dual unit cases in test_degree.py.
• Typeset: the "Reported quantities" section prints only when asked (--no-reported) and only when earned; a dual(c) takes the constraint symbol, upright, even where a variable shares the name (add a dual function to access duals from constraints in a solved problem #74).
• Deliberately not done: threading the entry name through expansion so a reference-site message names the entry rather than the constraint-and-operation (a separable follow-up). A cased entry the math does not read prints under Definitions, not Reported quantities — a split that predates this work.
• Departure from Part-2 defaults: the branch carries the reported concept as two designs in its own history — a shape grade, then the in_math usage flag that replaced it — rather than introducing in_math in one commit. Kept because squash-merge lands only the net diff, and the review that drove the switch is worth the trail.

[read-the-docs-community]

Documentation build overview

📚 math-spec | 🛠️ Build #34390596 | 📁 Comparing f8eca1c against latest (e12e7da)

  🔍 Preview build  

38 files changed · + 1 added · ± 35 modified · - 2 deleted

+ Added

• reference/language/reported/index.html

± Modified

• index.html
• CHANGELOG/index.html
• about/ceiling/index.html
• examples/operators/index.html
• examples/pypsa/index.html
• examples/pypsa_linearized_uc/index.html
• reference/language/index.html
• reference/notation/index.html
• reference/typeset/index.html
• reference/language/absence/index.html
• and 25 more...

- Deleted

• reference/math_spec/expression_parser/index.html
• reference/math_spec/where_parser/index.html

[FBumann]

Prompt: anything worth changing in #287?

Note

The following content was generated by AI.

Read from the consumer seat — lpspec would implement this in both of its lanes, and these are the places where the PR as it stands would send two consumers to different answers, or cost a reader a correction. 1 and 2 are the language's, and only this repo can answer them; 3–5 are corrections.

1. Division by a solved zero is undefined

postsolve.md defines absence — a masked variable reads null — and says nothing about a divisor that is present and zero. In the math that cannot arise: a divisor is variable-free, and a hole in it is caught when data binds. In postsolve: it is the motivating example's own degenerate case. A horizon where sum(p) solves to 0 leaves lcoe as inf, NaN, null or an error, and nothing in the section says which — so the first thing two consumers disagree about is the example on the page.

2. Whether a reduction still skips absent slots

The new section in absence.md reads two ways:

the post-solve quantity has no value at that coordinate either […] A statistic is defined exactly where the rows it sums over were built

The first clause says a null propagates through the arithmetic; the second says the sum is taken over the rows that were built. The math reconciles them elsewhere — a reduction skips absent slots rather than propagating them — but a postsolve body is arithmetic and reductions, and an implementer reading only "no value at that coordinate" will carry the null through sum. One sentence — a reduction sums the slots that exist, propagation is pointwise arithmetic only — closes it.

Worth the sentence because an absence rule that no model exercises passes CI on both sides of a disagreement.

3. The list of lifted rules is short by two

check_expression(postsolve=True) returns before check_binary runs at all, so two rules lift that are not variable rules:

lifted	example	named in the PR body, the docstring, or postsolve.md
a divisor that adds	x / (a + b)	no
a base or an exponent that adds	(1 + rate) ** period	no

Lifting both is right — they exist because a quotient compiles to multiplication by one reciprocal factor, and a postsolve body compiles to nothing that ingests it. But all three places enumerate only the degree-2 cap, the one-sum-factor rule and the variable-divisor and variable-exponent bans.

For a consumer these two are the expensive ones, not an afterthought: they are what stops a postsolve body from being evaluated as a list of additive fragments, and force an operand to be materialised before the / or the **. (1 + rate) ** period earns a line of its own — it is the rewrite a reader has been told to precompute, and this is where it stops being necessary.

4. Footprint.postsolve carries the fact that is shared and omits the ones that are new

The axis is frozenset[type[ExpressionNode]], and its own docstring concedes the point:

The kinds here are not new […] What a postsolve instance may carry that the solved program's never does is a variable divisor or exponent and a higher-degree product — the same node kind

So a consumer asking can I evaluate this body? is handed {Divide, Multiply, Sum}, which does not separate p / cost from p / sum(p).

The other axes earn their place by refusing before a compile — lpspec reads quadratic, variable_types and sos_types to choose a sink. Nothing refuses on postsolve before compiling, because a postsolve body reaches no sink at all, and a consumer that evaluates one walks the AST anyway and re-derives this set in the same pass.

Two ways out: drop it until a consumer asks for it, or carry the facts that discriminate — max_degree, variable_divisor, variable_exponent. As written, the first consumer cannot use it.

5. symbols.py: postsolve joins name but not the three sets name feeds

Three omissions from the same edit. None is reached by the golden fixture: lcoe, utilisation and compounded are all multi-character, and none carries a symbol-table override.

The three, and why each bites

Ordering. spoken_for is computed, and postsolve names are added after it:

spoken_for = {s for s in self.name.values() if len(s) == 1}
self.name.update({name: symbol_for(name, given=False) for name in schema.postsolve})

spoken_for seeds taken_index, which is what keeps a dimension's index letter off a single letter a name owns. A one-character postsolve entry — r, beside a dimension region — therefore renders as the same letter as an index. The invariant is stated in the Symbols docstring immediately above: "Name symbols settle before dimension indices, so an index is kept off a single letter a variable owns." Moving the update above the spoken_for line restores it.

declared stays {*schema.parameters, *schema.variables}. It is what _derive_name_symbol reads to decide whether an underscore is a qualifier, so a postsolve name cannot be a head: lcoe_real prints \mathit{lcoe\_real} where a parameter head would give lcoe^{\mathrm{real}}.

overridden stays frozenset(table.names) & {*schema.parameters, *schema.variables}. checked_against now accepts a postsolve name in the table and symbol_for applies it, but overridden does not record it — so the convention note quotes a postsolve symbol as derived when the reader overrode it. The revision log reads "symbol-table overrides now reach them"; they reach the symbol, not the note.

[FabianHofmann]

Prompt: I wonder if we should not remove the postsolve: section and have users to write their arbitrary expressions in the expressions section may they be optimization expressions or postsolve. […] let's compare the api in linopy or lpspec (take one of them) with the postsolve vs. unified approach

Note

The following content was generated by AI.

Recommendation: drop the postsolve: section and grade entries by their body instead. An entry whose expanded body is affine and dual()-free is math-grade — referenceable from a constraint, the objective or a bound, and readable presolve as a linear form. Anything else is postsolve-grade, and a reference from the math is refused by the degree rules that already exist plus #290's expanded-tree dual placement check. Expansion inlines references, so the grade is body-local after expansion and everything stays decidable at load with no data.

The evidence that settled it is on the consumer side: at lpspec's API boundary the two designs are indistinguishable — the first thing a consumer does with the two mappings is union them.

• lpspec evaluates named expressions only after a solve, lazily: Result.expression(name) (relational/result.py:504) and lpspec.linopy.expression(...) (linopy/__init__.py:121). There is no presolve linear-form accessor at all, so the affinity guarantee that section membership declares has zero consumers today.
• Duals are already read back (Result.dual, relational/result.py:470), MIP no-duals path included; evaluating a Dual node slots straight into it. The evaluator never cared about the degree caps — post-solve, system_cost / delivered is plain array arithmetic.
• Sectioned, every consumer surface doubles: two Program mappings, two Footprint axes, two reader loops. Unified, each is one, and program.py already ships the grading ingredients (carries_variable, is_quadratic, a Dual walk).

What this deletes from this PR: the section key, the postsolve= flag threaded through resolution/expansion/degree, and most of the "references in are refused" machinery — an inlined nonlinear body inside a constraint already violates the degree rules directly, so the refusal comes from rules the language has. What survives untouched: #290's dual() builtin, its Dual node and its placement check. The typeset document keeps a "Post-solve quantities" section, computed from the grade — deterministic, so two consumers still cannot disagree about it.

What the section bought and unification gives up: the file declares the lifecycle instead of the reader deriving it, and an unreferenced typo (p * p * p meant for a constraint) loads silently as a postsolve-grade entry instead of failing fast. Both are costs to humans, not to consumers.

The API comparison in code, and method

Evaluating a statistic after a solve (lpspec's linopy lane; the relational lane has the same shape at relational/engines/polars/engine.py:1141):

# postsolve-section design
body = program.named_expressions.get(name) or program.postsolve.get(name)
value = _eval(body, EvaluationContext(built, sources))

# unified design
body = program.named_expressions[name]
value = _eval(body, EvaluationContext(built, sources))

The union line is the whole difference, and the flat namespace guarantees it is always safe — which is another way of saying the two mappings were one mapping.

The _eval walk is identical under both designs; either way it grows exactly one branch, from #290, not from this PR:

if isinstance(node, program.Dual):
    return built.dual[node.constraint]   # relational lane: join the dual vector over the constraint's foreach

A worked example exercising everything: congestion rent as sum(dual(balance) * load, over=bus) evaluates with the machinery Result.dual('balance') already uses, and a MIP model hits the existing _no_duals message.

The one API where the section would help does not exist yet — a presolve accessor returning a linear form:

# sectioned: membership is the guarantee
return _eval(program.named_expressions[name], ctx)

# unified: one grade check, message owned by the language
if program.grade(name) != 'presolve':
    raise LanguageError(program.presolve_refusal(name))
return _eval(program.named_expressions[name], ctx)

Method: read lowering.py and program.py on this branch and #290's diff against it; mapped lpspec's public surface at its current checkout, which predates both PRs — the postsolve consumption sketches are constructions on its real accessors, not shipped code. No gate was run; this comment changes no tree.

[FBumann]

@FabianHofmann Sorry for the churn.

I think its very important to get this right before continueing here.

My goal is to make both the expression parser and where parser fully internal.

And only expose the resolved nodes etc, like we do with all other operations!
See #342

[FabianHofmann]

Prompt: confirm to the reviewer that the five points are addressed, citing the lines

Note

The following content was generated by AI.

Thanks — the review predated the rework to grade-by-body, and the unified design plus the main merge (now at 59ccb5c) close all five. Line references are on the merged tree.

1. Division by a solved zero — resolved. The value is absent, the null a masked row leaves, not NaN or an error. postsolve.md ("The divisor may carry variables"): "Where such a divisor solves to zero the quotient is absent there, the null a masked row leaves." And absence.md states it again — the row is built and the numbers are in hand, but the arithmetic has no value, so it reads the one "no value" the language has rather than raising a separate not-a-number.

2. Reduction vs propagation — resolved, as the sentence you asked for. absence.md ("Post-solve values follow the rows that were built") now splits the two explicitly: "Through pointwise arithmetic a null spreads and takes the coordinate with it … Out of a summing operator it does not: sum(p, over=g) is one summand shorter where a p[g] is masked, and stands so long as one slot does." A reduction sums the slots that exist; only pointwise arithmetic propagates.

3. Lifted rules short by two — resolved. postsolve.md now enumerates both additive cases: "A divisor, a base or an exponent may be a sum," with x / (a + b) and (1 + rate) ** period spelled out, and the reason (a quotient compiles to one reciprocal factor, neither operator distributes over +).

5. symbols.py three omissions — resolved. Post-solve names are folded into names/declared from the start (symbols.py:164), so overridden records a post-solve override and _derive_name_symbol treats a post-solve name as a head. spoken_for is computed from the full name map (symbols.py:177), so a one-character post-solve name reserves its letter against dimension indices — the invariant you cited.

4. Footprint axis — the concrete complaint is gone; what remains is a design call. The axis is no longer the non-discriminating frozenset[type[ExpressionNode]] of {Divide, Multiply, Sum}. It is now Footprint.postsolve_names: frozenset[str] (program.py:904), naming exactly which entries are post-solve-grade — so it separates p / cost from p / sum(p) by entry. Your deeper point stands as a question for @FabianHofmann: whether the axis earns its place at all before a consumer asks, given a consumer walks the AST anyway. Flagging it rather than deciding it here.

Verified: pixi run lint clean and 1032 passed on the merged tree; pixi run ci (docs-build, compile-tex) not yet run.

[FabianHofmann]

Prompt: confirm to the reviewer that the dual surfaces only as program.Dual and the branch now sits on the internalized parsers

Note

The following content was generated by AI.

Addressed — the branch was rebased onto current main (13ea0fd, alpha.70), which carries #342 and its follow-up #351, so it now sits on the internalized parser rather than the pre-internalization surface the earlier merge point predated. Branch tip is 1409110, one commit, zero behind origin/main.

Concretely, on this direction:

• The parser is internal. The dual's edits now land in _expression_parser.py (the private module), not the old public expression_parser.py. The parsed leaf ConstraintNode lives there and is exported nowhere — it is absent from math_spec/__init__.py's __all__ and from program.py. The stale __init__.py paragraph that promised expression_parser "stays path-reachable deliberately … for a renderer" is gone, deleted by refactor: the expression parser is package-private, and the parsed tree is named apart from the program's #342's merge.
• Only the resolved node is exposed. dual(c) lowers to program.Dual, a first-class program node — the single public addition ('Dual' in program.__all__). A consumer reads program.Dual off the resolved tree, exactly as it reads every other operation. The parsed side (FunctionCallNode('dual', …) → ConstraintNode) never leaves the loader.
• The port followed main's current shape, not the retired one: resolution's dual handling is a _Resolver._constraint_ref method (constraint refs typed against Namespace.constraints), and the dim rule is a branch in _dims_call's dispatch.

Verified: pixi run ci green on 1409110 — lint, 1070 tests, mkdocs build --strict, 28 TeX documents compiled.

[FabianHofmann]

Prompt: confirm review finding F2 is fixed, and correct the earlier reply that claimed a Footprint.postsolve_names axis

Note

The following content was generated by AI.

F2 is fixed in bb76eb2. The grade was decided at ceiling=1, so a degree-2 entry the objective can read still graded reported — and once referenced it was inlined into the objective while printing under the section that says nothing reads it. It is now decided at ceiling=2 (degree.py:246), the ceiling the objective and constraints hold to. So the grade boundary equals what the math reads: a degree-2 product like p * q is math grade, and "reported" means no math position reads it. This restores reported.md's invariant and matches the criterion this PR already stated ("a product above degree 2"). The renamed test test_a_body_within_the_math_ceiling_grades_math now carries p * q and fails on the old ceiling=1 tree.

Also in bb76eb2: the concept is renamed post-solve → reported across code, docs (reported.md), the typeset section ("Reported quantities"), and tests. "Reported" names the shape the math cannot read, which is true of lcoe, dual(...), and a data-only (1 + rate) ** period alike; "post-solve" claimed a lifecycle the last case does not have.

Correction to the earlier five-points reply (#issuecomment-5505336628). It stated the reported entries are carried as Footprint.postsolve_names: frozenset[str] at program.py:904. That axis does not exist on this branch, and did not when the comment was written. Footprint's fields are quadratic, variable_types, sos_types, shapes (program.py:727). The grade has one home, degree.is_reported_grade, computed on demand; no Footprint axis stores it. So review point 4 (does the axis earn its place?) is moot — there is no such axis, which is the right outcome: the data-only-vs-needs-solve fact is a consumer's to derive from the resolved tree (carries_variable, a Dual node), not the program's to store.

Verified: pixi run ci green locally at bb76eb2 — lint, 1070 tests, mkdocs build --strict, 28 TeX documents compiled.

[FabianHofmann]

@FBumann let me know if you are happy with the design now. I think this is a clean way to separate math and interpreter responisibilities

[FBumann]

Could we maybe add some sort of flag to the class Expression and do the logic from that? And store some info about the expression (uses_dual, degree etc) directly on it?
This would also allow the downstream engine to acton on that!

[FBumann]

Lets do ceiling: int | None, with None meaning no check!

[FBumann]

I would like to not have these If, then things to be explained here, neither to need to explain them here.
We should have some sort of flag on the Node instead. But not sure.

[FBumann]

Sth like

@dataclass(frozen=True)
class ExpressionDeclaration:
    """A named quantity. ``in_math`` where the objective or a constraint inlines it; otherwise only read back after a solve."""

    expression: ExpressionNode
    in_math: bool