refactor(language): the front end is a package, and the fence is a test

docs/ARCHITECTURE.md

@@ -25,8 +25,8 @@ below protects it.
 
 ```mermaid
 flowchart TB
-    Y[YAML file] -->|"parse + validate<br/>(schema.py, validation.py)"| MS[MathSchema]
-    MS -->|"expand macros: / expressions: (expansion.py)<br/>expand piecewise: blocks (piecewise.py)<br/>resolve names to typed nodes (resolution.py)<br/>check dim sets (dimensions.py)<br/>— backends never see any of them"| AST["core AST<br/>= the only contract between layers<br/>fully typed: names resolved, dims checked"]
+    Y[YAML file] -->|"parse + validate<br/>(language/schema.py, language/validation.py)"| MS[MathSchema]
+    MS -->|"expand macros: / expressions: (language/expansion.py)<br/>expand piecewise: blocks (piecewise.py)<br/>resolve names to typed nodes (language/resolution.py)<br/>check dim sets (language/dimensions.py)<br/>— backends never see any of them"| AST["core AST<br/>= the only contract between layers<br/>fully typed: names resolved, dims checked"]
     AST -->|"api.py: check / build / solve / write"| LOWER
     AST -.->|"lpspec.linopy<br/>(opt-in shim: build / extend)"| BUILD
     LOWER -->|"outside the language:<br/>LanguageError naming the construct"| ERR["load error<br/>(no fallback)"]
@@ -171,7 +171,12 @@ that made an implementation choice load-bearing in the language's rulebook.
    and the plan/query/xarray are backend-private. The AST crossing that seam is **fully
    resolved** — names are typed `Variable`/`Parameter`/`Dimension` nodes — so a backend cannot
    hold its own opinion about what a name refers to. Resolving independently is
-   how the two lanes silently disagreed about scoping before.
+   how the two lanes silently disagreed about scoping before. The waist is
+   closed from the front too: nothing under `src/lpspec/language/` imports
+   `lowering`, `piecewise`, `sources`, `api` or any consuming subpackage, so
+   what a model *means* cannot depend on what is done with it. That is the
+   mirror of rule 2 and it is enforced the same way, off the path
+   (`LANGUAGE_MAY_IMPORT`).
 2. **The engine knows nothing about linopy, xarray or YAML.**
    `src/lpspec/relational/` goes polars → highspy → solver, with linopy's
    semantics as a spec to match rather than code to share; it never sees the
@@ -271,21 +276,21 @@ than discovered at solve time.
 
 | Module | Role |
 |---|---|
-| `_yaml.py` | the only place a file is read: YAML 1.2 booleans, duplicate keys refused |
-| `schema.py` | pydantic schema incl. `expressions:` / `macros:` / `piecewise:` |
-| `expression_parser.py`, `where_parser.py` | text → core AST; grammar only, dependency-free |
-| `expansion.py` | named-expression / macro substitution (pre-dispatch) |
-| `resolution.py` | one flat namespace; `NameNode` → typed `Variable`/`Parameter`/`Dimension` nodes |
-| `dimensions.py` | static dim-set checking over the resolved AST |
-| `validation.py` | load-time: parse, expand, resolve, check everything |
+| `language/_yaml.py` | the only place a file is read: YAML 1.2 booleans, duplicate keys refused |
+| `language/schema.py` | pydantic schema incl. `expressions:` / `macros:` / `piecewise:` |
+| `language/expression_parser.py`, `language/where_parser.py` | text → core AST; grammar only, dependency-free |
+| `language/expansion.py` | named-expression / macro substitution (pre-dispatch) |
+| `language/resolution.py` | one flat namespace; `NameNode` → typed `Variable`/`Parameter`/`Dimension` nodes |
+| `language/dimensions.py` | static dim-set checking over the resolved AST |
+| `language/helpers.py` | the closed set of built-in operators: their *names* and *call shapes* — no registry |
+| `language/validation.py` | load-time: parse, expand, resolve, check everything |
 | `piecewise.py` | `piecewise:` → λ-formulation declarations + curvature guard |
 | `api.py` | native entry point: `check` / `build` / `solve` / `write`, linopy-free |
 | `typeset/` | **spike** — resolved AST → LaTeX / Typst / Markdown. A reader, not a lane: no model, no data, no plan ([README](https://github.com/FBumann/lpspec/blob/main/src/lpspec/typeset/README.md)) |
 | `__main__.py` | `python -m lpspec <format>` — a shell front for the verbs that bind no data |
 | `sources.py` | bind runtime data (parquet paths / in-memory tables) to a validated schema |
 | `lowering.py` | core AST → logical plan (defines the relational subset) |
-| `helpers.py` | the closed set of built-in operators: their *names* and *call shapes* — no registry |
-| `errors.py` | the exception hierarchy; the one module the engine may import |
+| `errors.py` | the exception hierarchy; the one module either fenced side may import |
 | `relational/plan.py` | frozen logical-plan dataclasses |
 | `relational/frames.py` | the boundary — caller tables in, via the Arrow PyCapsule protocol |
 | `relational/compiler.py` | plan → lazy frames; pure, reads nothing |
@@ -300,13 +305,23 @@ than discovered at solve time.
 | `linopy/builder.py` | eager backend: core AST → `linopy.Model` |
 | `linopy/semantics.py` | where this lane answers linopy's v1 arithmetic convention — one home, as linopy's own `semantics.py` is |
 
-Two subpackages, and the directory *is* the rule in both cases. Everything
-under `relational/` is the engine and imports nothing else from the package;
-everything under `linopy/` is the opt-in eager lane and is the only code
-allowed to import linopy or xarray; everything under `typeset/` reads the AST
-and writes text, and reaches neither the plan nor any data. `tests/test_architecture.py` reads
-membership off the path, so neither fence can be stepped over by naming a
-file differently.
+**Four subpackages, and the directory *is* the rule in every case.** Everything
+under `language/` produces the AST and may not reach a consumer of it;
+everything under `relational/` is the engine and imports nothing else from the
+package; everything under `linopy/` is the opt-in eager lane and is the only
+code allowed to import linopy or xarray; everything under `typeset/` reads the
+AST and writes text, and reaches neither the plan nor any data.
+`tests/test_architecture.py` reads membership off the path, so no fence can be
+stepped over by naming a file differently.
+
+`language/` and `relational/` are the two halves of the waist and their fences
+point the same way — outward, at `errors.py`, the one leaf both may import. The
+modules left at the top level are the ones that are legitimately *both*:
+`lowering.py` reads the AST and writes the plan, `piecewise.py` emits
+declarations but consults the subset test to do it, `sources.py` binds data to
+a validated schema, and `api.py` runs the lot. A module that belongs to neither
+side is a module that has to sit on the line, and the flat namespace is where
+it sits.
 
 ### Naming across the layers
 
@@ -316,7 +331,7 @@ suffix**, which is what keeps the three vocabularies from colliding:
 
 | Layer | Suffix | Example |
 |---|---|---|
-| YAML block (`schema.py`) | `Block` | `VariableBlock`, `PiecewiseBlock` |
+| YAML block (`language/schema.py`) | `Block` | `VariableBlock`, `PiecewiseBlock` |
 | Core AST (`*_parser.py`) | `Node` | `VariableNode`, `DimensionComparisonNode` |
 | Logical plan (`relational/plan.py`) | none / `Declaration` | `Variable`, `VariableDeclaration` |
 
@@ -364,7 +379,7 @@ lowering case → differential test on both sinks → SPEC §5/§7, and this fil
 structural.
 
 Two things are deliberately *not* per-primitive work, because they are one
-implementation each: a primitive's dim rule lives only in `dimensions.py` —
+implementation each: a primitive's dim rule lives only in `language/dimensions.py` —
 both its dim *set* and its verdict on an operand that lacks the dim being
 reduced along, which lowering asks for rather than deciding again — and the
 dense-label assignment that gives a coordinate its solver index lives only in

examples/walkthrough.py

@@ -27,8 +27,8 @@
 import polars as pl
 
 import lpspec as lps
-from lpspec.expansion import parse_and_expand
-from lpspec.expression_parser import parse_expression
+from lpspec.language.expansion import parse_and_expand
+from lpspec.language.expression_parser import parse_expression
 from lpspec.lowering import lower_program
 from lpspec.relational.executor import PolarsExecutor
 from lpspec.sources import tidy_sources

pyproject.toml

@@ -165,7 +165,7 @@ extend-fixable = ["B", "SIM", "RUF", "C4", "UP"]
 "bench/**/*.py" = ["T20"]  # a harness reports by printing; stdout is its interface
 "tools/**/*.py" = ["T20"]  # same: a generator says what it wrote
 "tests/golden/__main__.py" = ["T20"]  # likewise a generator, and only reachable as a command
-"src/lpspec/where_parser.py" = ["N806"]  # grammar tokens (NOT/AND/OR) are named after keywords
+"src/lpspec/language/where_parser.py" = ["N806"]  # grammar tokens (NOT/AND/OR) are named after keywords
 
 [tool.ruff.format]
 quote-style = "single"

src/lpspec/__init__.py

@@ -27,7 +27,7 @@
     PiecewiseExpansionError,
     SchemaError,
 )
-from lpspec.schema import MathSchema
+from lpspec.language.schema import MathSchema
 from lpspec.typeset import SymbolTable, to_latex, to_markdown, to_typst
 
 __all__ = [

src/lpspec/api.py

@@ -31,13 +31,13 @@
 from pathlib import Path
 from typing import TYPE_CHECKING, Any
 
-from lpspec._yaml import read_yaml
+from lpspec.language._yaml import read_yaml
+from lpspec.language.schema import MathSchema
+from lpspec.language.validation import validate_expressions
 from lpspec.lowering import lower_program
 from lpspec.piecewise import expand_piecewise
 from lpspec.relational.executor import PolarsExecutor, Result
-from lpspec.schema import MathSchema
 from lpspec.sources import tidy_sources
-from lpspec.validation import validate_expressions
 
 if TYPE_CHECKING:
     from collections.abc import Mapping

src/lpspec/language/__init__.py

@@ -0,0 +1,21 @@
+"""The language: what a YAML file may say, and what it means.
+
+Everything from the bytes on disk to a fully typed, dim-checked core AST —
+the file reader, the schema, the two grammars, expansion, resolution, the dim
+rules, and the load-time pass that runs them all. The AST this package
+produces is the narrow waist of docs/ARCHITECTURE.md: everything downstream
+reads it, and nothing downstream is visible from here.
+
+**The directory is the rule, in the direction the engine's is not.** Hard rule
+2 says the engine never sees the schema or the AST; this is its mirror —
+nothing under ``language/`` may import ``lowering``, ``piecewise``,
+``sources``, ``api``, or any of the three consuming subpackages. What a model
+*means* cannot depend on what any consumer does with it, which is what makes
+``lps.check()`` a pass with no data and no plan, and a second consumer cheap.
+``errors.py`` stays outside deliberately: it is the dependency-free leaf both
+this package and the engine may import (``ENGINE_MAY_IMPORT``), and moving it
+in would put the language's path on the engine's import list.
+
+``tests/test_architecture.py`` reads membership off the path, so a new
+front-end module cannot land outside the fence by being spelled differently.
+"""

src/lpspec/dimensions.py → src/lpspec/language/dimensions.py

@@ -40,7 +40,7 @@
 from typing import TYPE_CHECKING, assert_never
 
 from lpspec.errors import DimensionError
-from lpspec.expression_parser import (
+from lpspec.language.expression_parser import (
     ArithmeticNode,
     BinaryOperatorNode,
     ComparisonNode,
@@ -55,8 +55,8 @@
     UnaryOperatorNode,
     VariableNode,
 )
-from lpspec.resolution import Namespace, expression_of, where_of
-from lpspec.where_parser import (
+from lpspec.language.resolution import Namespace, expression_of, where_of
+from lpspec.language.where_parser import (
     AndNode,
     BooleanLiteralNode,
     DimensionComparisonNode,
@@ -73,7 +73,7 @@
 if TYPE_CHECKING:
     from collections.abc import Mapping, Sequence
 
-    from lpspec.schema import MathSchema
+    from lpspec.language.schema import MathSchema
 
 
 def dims_of(

src/lpspec/expansion.py → src/lpspec/language/expansion.py

@@ -40,7 +40,7 @@
 from typing import TYPE_CHECKING, assert_never, overload
 
 from lpspec.errors import SchemaError
-from lpspec.expression_parser import (
+from lpspec.language.expression_parser import (
     ArithmeticNode,
     BinaryOperatorNode,
     ComparisonNode,
@@ -60,7 +60,7 @@
 if TYPE_CHECKING:
     from collections.abc import Callable
 
-    from lpspec.schema import MacroBlock, MathSchema
+    from lpspec.language.schema import MacroBlock, MathSchema
 
 #: Backstop against pathological nesting the cycle check cannot see.
 _MAX_DEPTH = 50

src/lpspec/resolution.py → src/lpspec/language/resolution.py

@@ -20,8 +20,8 @@
 from typing import TYPE_CHECKING, assert_never
 
 from lpspec.errors import LanguageError
-from lpspec.expansion import parse_and_expand
-from lpspec.expression_parser import (
+from lpspec.language.expansion import parse_and_expand
+from lpspec.language.expression_parser import (
     ArithmeticNode,
     BinaryOperatorNode,
     ComparisonNode,
@@ -36,8 +36,8 @@
     UnaryOperatorNode,
     VariableNode,
 )
-from lpspec.helpers import BUILTINS, EDGE_WRAP, call_shape_error, edge_error, unknown_helper_message
-from lpspec.where_parser import (
+from lpspec.language.helpers import BUILTINS, EDGE_WRAP, call_shape_error, edge_error, unknown_helper_message
+from lpspec.language.where_parser import (
     AndNode,
     BooleanLiteralNode,
     DimensionComparisonNode,
@@ -55,7 +55,7 @@
 if TYPE_CHECKING:
     from collections.abc import Iterable, Mapping
 
-    from lpspec.schema import MathSchema
+    from lpspec.language.schema import MathSchema
 
 
 class Namespace:

src/lpspec/schema.py → src/lpspec/language/schema.py

@@ -7,7 +7,7 @@
 
 from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
 
-from lpspec.helpers import BUILTIN_NAMES
+from lpspec.language.helpers import BUILTIN_NAMES
 
 if TYPE_CHECKING:
     from collections.abc import Iterable

src/lpspec/validation.py → src/lpspec/language/validation.py

@@ -23,10 +23,10 @@
 from types import MappingProxyType
 from typing import TYPE_CHECKING, assert_never
 
-from lpspec.dimensions import check_schema
 from lpspec.errors import SchemaError
-from lpspec.expansion import expand, parse_and_expand, parse_template
-from lpspec.expression_parser import (
+from lpspec.language.dimensions import check_schema
+from lpspec.language.expansion import expand, parse_and_expand, parse_template
+from lpspec.language.expression_parser import (
     ArithmeticNode,
     BinaryOperatorNode,
     ComparisonNode,
@@ -40,14 +40,14 @@
     UnaryOperatorNode,
     VariableNode,
 )
-from lpspec.helpers import BUILTINS, unknown_helper_message
-from lpspec.resolution import Namespace, resolve_expression, resolve_where
-from lpspec.where_parser import parse_where
+from lpspec.language.helpers import BUILTINS, unknown_helper_message
+from lpspec.language.resolution import Namespace, resolve_expression, resolve_where
+from lpspec.language.where_parser import parse_where
 
 if TYPE_CHECKING:
     from collections.abc import Mapping, Sequence
 
-    from lpspec.schema import MathSchema
+    from lpspec.language.schema import MathSchema
 
 
 def validate_expressions(

src/lpspec/linopy/__init__.py

@@ -47,8 +47,10 @@
 
 
 from lpspec._notes import note
-from lpspec._yaml import read_yaml
 from lpspec.errors import LanguageError
+from lpspec.language._yaml import read_yaml
+from lpspec.language.schema import MathSchema
+from lpspec.language.validation import validate_expressions
 from lpspec.linopy.builder import build_model
 from lpspec.linopy.loader import (
     build_dim_coords,
@@ -57,8 +59,6 @@
     load_parameters,
 )
 from lpspec.piecewise import expand_piecewise, validate_piecewise_data
-from lpspec.schema import MathSchema
-from lpspec.validation import validate_expressions
 
 # **This lane speaks v1, and the option is global, so importing sets it.**
 #

src/lpspec/linopy/builder.py

@@ -17,7 +17,7 @@
 
 from lpspec._notes import note
 from lpspec.errors import DataError, LanguageError, null_bounds_message
-from lpspec.expression_parser import (
+from lpspec.language.expression_parser import (
     ArithmeticNode,
     BinaryOperatorNode,
     ComparisonNode,
@@ -31,11 +31,9 @@
     UnaryOperatorNode,
     VariableNode,
 )
-from lpspec.helpers import EDGE_WRAP, unknown_helper_message
-from lpspec.linopy import semantics
-from lpspec.linopy.loader import check_divisors_cover
-from lpspec.resolution import Namespace, expression_of, where_of
-from lpspec.where_parser import (
+from lpspec.language.helpers import EDGE_WRAP, unknown_helper_message
+from lpspec.language.resolution import Namespace, expression_of, where_of
+from lpspec.language.where_parser import (
     AndNode,
     BooleanLiteralNode,
     DimensionComparisonNode,
@@ -48,14 +46,16 @@
     VariableDefinedNode,
     WhereNode,
 )
+from lpspec.linopy import semantics
+from lpspec.linopy.loader import check_divisors_cover
 
 if TYPE_CHECKING:
     from collections.abc import Callable, Hashable, Mapping
 
     import linopy
     import pandas as pd
 
-    from lpspec.schema import MathSchema
+    from lpspec.language.schema import MathSchema
 
 # Mapping from YAML comparison operators to linopy sign strings
 _SIGN_MAP = {'==': '=', '<=': '<=', '>=': '>='}

src/lpspec/linopy/loader.py

@@ -9,7 +9,7 @@
 import xarray as xr
 
 from lpspec.errors import DataError, duplicate_coordinate_message, sparse_divisor_message
-from lpspec.expression_parser import (
+from lpspec.language.expression_parser import (
     BinaryOperatorNode,
     ComparisonNode,
     FunctionCallNode,
@@ -20,7 +20,7 @@
 )
 
 if TYPE_CHECKING:
-    from lpspec.schema import MathSchema
+    from lpspec.language.schema import MathSchema
 
 
 def build_master_coords(