Architecture: the typed AST is the narrow waist (producers / consumers / transformers)

[FBumann]

Summary

The typed expression AST (expression_parser + where_parser dataclass nodes,
together with the schema) is not just a parser internal — it is the narrow
waist of the whole system. Today it has one producer (YAML) and two consumers
(the eager builder, the relational lowering/IR). Both backends are already
just consumers of a shared, solver-agnostic algebraic model.

Recognising this reframes the project: it is not "YAML → linopy", it is a
portable model AST with a YAML front-end, N execution backends, and M
renderers / analyzers / transformers. That is a stronger and more durable
architecture, and it changes how we should scope several roadmap items
(LaTeX export, the eager-vs-relational router, piecewise/SOS support).

This issue records the finding and the design rules that follow from it.

Where it came from

Two prior findings led here:

1. linopy 0.9 has moved past the flat A-matrix. It now ships
   add_piecewise_formulation, add_sos_constraints,
   add_indicator_constraints, semi_continuous, and a reformulation layer
   (reformulate_sos_constraints / apply_sos_reformulation / dualize).
   These are higher-level modeling constructs; the relational backend, which
   bypasses linopy.Model and streams (col, row, coeff) to the solver,
   inherits none of them, while the eager builder (a pure consumer of linopy)
   inherits them for free. The two backends are diverging in capability.

2. LaTeX export forced the layering question. Symbolic LaTeX (\sum_{g} p_g c_g) must be rendered from the AST, never the IR — because the IR has
   already thrown away exactly what LaTeX needs (sum(over=…) → GROUP BY,
   where → row absence, names → integer labels; this is the readable-vs-
   functional round-trip already noted in SPEC Q8). The moment you say "LaTeX
   is a consumer of the shared AST", a whole family of consumers falls out.

The consumer taxonomy

Consumers cluster into four kinds, and the layer rule is uniform:

Consumer	Kind	Layer	Needs data?
Symbolic LaTeX / MathML / Markdown-math	renderer	AST	no
Natural-language "explain"	renderer	AST	no
Graphviz (var↔constraint / param→var DAG)	renderer	AST	no
AMPL / GAMS / Pyomo / JuMP emitter	renderer	AST	no
Shape / dimension inference	analyzer	AST	no
Unit / dimensional-consistency checker	analyzer	AST + units meta	no
Size / sparsity estimator (nnz, nrows)	analyzer	AST	no
Model diff (structural, for extend())	analyzer	AST	no
Reformulation (piecewise/SOS → aux vars)	transformer	AST→AST	no
Symbolic presolve / constant-folding	transformer	AST→AST	no
LSP / language server (hover, go-to-def)	tooling	AST	no
Instantiated LaTeX (concrete rows)	sink	IR	yes
LP / MPS	sink	IR	yes

Layer rule: renderers and analyzers branch at the AST and never touch the
IR; transformers run before lowering (AST→AST); only data-bearing,
fully-expanded outputs are IR sinks. Same reason each time — the IR has already
discarded names, quantifiers, and where-structure.

Load-bearing consumers (not just nice-to-have)

• Size / sparsity estimator. This is the cost model the automatic
  eager-vs-relational router needs. Reads schema dim-sizes + where-strings,
  estimates nrows/ncols/nnz without building anything. The router is not
  optional glue — it is a consumer the architecture requires, and it lives
  purely at the AST.

• AST-level reformulation. The clean fix for the piecewise/SOS divergence
  in finding (1): if a piecewise:/sos: block reformulates at the AST into
  aux-variable declarations + linking constraints, both backends inherit it
  automatically — no reimplementing big-M in SQL, no eager/relational
  capability gap. Highest-leverage item here.

• Shape / dimension inference. Both backends already do broadcast reasoning
  implicitly. Factoring it out as one shared analyzer removes duplication and
  simultaneously powers the size estimator, the unit checker, the LSP, and
  better load-time errors.

The recurring boundary: custom Python helpers

Every consumer hits the same ragged edge. The closed built-in language
(sum, group_sum, roll, arithmetic, comparisons) is universally
consumable — renderable, analyzable, reformulable. A registered custom Python
helper is opaque to every consumer, so each must reject-or-placeholder it.

Design pressure follows: every built-in added to the closed vocabulary lights
up all M consumers for free; every custom-helper escape hatch stays dark. This
is a concrete argument to grow the built-in vocabulary rather than lean on
custom helpers.

Second producer

The waist has a second potential producer, not just consumers. SPEC Q8 /
issue #3 (intercept add_variables/add_constraints to synthesise schema) is
exactly "a Python DSL that feeds the same AST". Under this framing it stops
being a curiosity and becomes "add a second front-end" — symmetric with adding
a renderer.

Proposed actions

• Add an "AST as narrow waist" section to SPEC (producer / consumer /
  transformer taxonomy; the AST→IR loses-readability boundary; the
  closed-language vs custom-helper rule).
• Extract shape/dimension inference as a shared AST analyzer (unblocks
  several consumers below).
• Build the size/sparsity estimator and wire it into automatic
  eager-vs-relational backend selection.
• Formalise AST-level reformulation as the mechanism for
  piecewise/SOS/indicator, so both backends stay in sync (ties to the
  linopy-0.9 finding).
• Then: symbolic LaTeX and AMPL/GAMS emitters as thin renderers on
  the same rails.

Related

• SPEC §12 (relational backend), SPEC §10 (piecewise/SOS "out of scope in v1"),
  SPEC Q8 (readable-vs-functional round-trip).
• Roadmap: complete .yaml representation #3 (Roadmap: complete .yaml representation) — the "second producer" angle.

[FBumann]

Closing: the load-bearing items landed or were overtaken, and the framing outlived the two-backend world it was written for.

• Shape / dimension inference as a shared analyzer — done. dimensions.py is a load-time pass over the resolved core AST, "so both lanes get the same answer by construction rather than by differential test" (feat!: static dim checking — the type is a set of dim names #68, fix: dim checking runs on both lanes, and binary ops union #70).
• AST-level reformulation for piecewise/SOS — done, at schema level rather than AST level, which is stronger: piecewise.py expands into ordinary affine declarations before anything is built, so both lanes receive identical schemas. "Formulations never enter the plan as expression nodes."
• Size / sparsity estimator — obsolete as scoped. It existed to feed the automatic eager-vs-relational router, and feat!: streaming-only — YAML is the product, linopy demoted to compat/oracle #26 removed the router by making streaming the only runtime lane; the eager lane is now the compat shim and the oracle.
• The custom-helper ragged edge — resolved by decision. Helpers are a closed set with no registry, so there is no opaque-to-every-consumer case left; unsayable math goes to a declared escape: island (escape: bounded, visible, terminal Python islands for unsayable constraints #38) instead.
• The "AST as narrow waist" SPEC section — never written, and the doc set has since been consolidated and cut by two thirds (docs: consolidate the doc set and cut it by two thirds #87). The phrase appears nowhere in the docs, and re-adding a taxonomy section now would fight that.

Genuinely live and tracked nowhere else: symbolic LaTeX / AMPL / GAMS renderers. Worth a fresh issue scoped to a renderer if one is wanted — the layer rule this issue established (renderers read the AST, never the IR, because lowering has already discarded names, quantifiers and where-structure) is the part to carry forward.