Roadmap

[ocots]

CTParser Roadmap v1

Last updated: July 7, 2026

Context

CTParser (v0.8.15) turns the @def DSL into executable model-building code, through two parsing backends living side by side in onepass.jl (~1500 lines): :fun generates function-based CTModels code, :exa generates a full ExaModels transcription (variables on a grid, scheme-dependent dynamics defects, cost quadrature, constraints). The package is a flat module (4 files: defaults.jl, utils.jl, onepass.jl, initial_guess.jl), with generated code addressed through mutable prefix Refs (PREFIX_FUN / PREFIX_EXA).

This roadmap is the companion of the CTDirect roadmap v1: its single most structural item — moving the discretization out of the parser (§2) — is the CTParser side of CTDirect roadmap §5 ("Own the ExaModels transcription in CTDirect"). The guiding principle:

The parser describes the problem; it does not discretize it. Scheme-dependent code (defects, quadrature, endpoint-control fixes, grids) belongs to CTDirect.

The rest re-scopes the open issues into: ExaModels 0.11 migration (§1), "1-D = scalar" simplification (§3), prefix removal (§4), parsing robustness (§5), syntax extensions (§6), and the usual architecture/tests/docs alignment with the Handbook (§7–§9).

Current pain points (evidence)

• The :exa backend is a second discretizer: p_dynamics_exa! hard-codes if scheme == :euler … :midpoint … :trapeze chains in generated code (onepass.jl:918-941, 1022-1035), p_lagrange_exa! duplicates the quadrature per scheme (1095-1103) — a parallel, restricted (4 schemes, uniform grid) implementation of what CTDirect does.
• Global mutable prefixes (PREFIX_EXA = Ref(…), prefix_exa!) route generated code to backend modules — fragile global state, #224.
• onepass.jl is a 1500-line monolith mixing the pattern-matching front-end, both codegen backends, and the @def entry points; header TODO comments (# - exa: …) accumulate.
• Known bugs/fragilities catalogued in #197 (match-pattern typos control_fun vs :control_fun, as_range returning a Vector where an expression-level range is expected, __wrap scoping).
• Silent mis-parses: leftover x/u symbols after replace_call don't error (#193); x(t0) with t0 undefined in the calling scope parses without a clear error (#139); slice notation support is partial (#260, PR #251 marks end-keyword and variable-range cases broken).
• ExaModels compat pinned to 0.9 (test env); 0.11 deprecates part of the API the generated code calls.
• Tests are flat scripts (test_onepass_*.jl, *_bis.jl duplicates) not following the Handbook template; docs are a single index.md on plain Documenter.

---

1. ExaModels 0.11 migration

Status: to do first — it is a compat blocker for the whole ecosystem (CTDirect and CTSolvers also pin ExaModels = "0.9").

The generated :exa code calls ExaModels.variable, constraint, constraint!, objective, solution, multipliers, plus the ExaCore/ExaModel lifecycle (through the prefix indirection). Work items:

1. Audit every ExaModels call site in generated code (onepass.jl: p_variable_exa!, p_state_exa!, p_control_exa!, p_constraint_exa!, p_dynamics_exa!, p_lagrange_exa!, p_mayer_exa!, and the solution getter block around onepass.jl:1426-1434) against the 0.11 API; replace deprecated forms.
2. Bump compat to ExaModels = "0.11" in test/Project.toml here, and coordinate the bump in CTDirect / CTSolvers / OptimalControl (one PR train, since the generated code and the caller must agree).
3. Run the full :exa test suite (test_onepass_exa*.jl, test_dynamics_exa.jl, test_exa_linalg.jl) + CTDirect's Exa problems as the cross-package check; GPU smoke test if available.
4. Capture in a short note which 0.11 changes affect the generated-code shape (e.g. constraint aggregation, multiplier retrieval) — input for §2, since the repatriation design (CTDirect §5) should target the 0.11 API directly, not the deprecated one.

---

2. Move the discretization out of the parser

Status: the structural item; joint with CTDirect roadmap §5 — the decision (function tracing vs expression-level IR) is made in the CTDirect §3/§5 design notes, this section is the CTParser side. Absorbs: #65 (non-uniform grid for exa), #161 (endpoint-control constraints per scheme), #134 (exa multipliers for boundary/path constraints).

Goal: the :exa parsing backend stops generating scheme-dependent transcription code. What remains in CTParser depends on the route chosen:

• Route 1 (function tracing, preferred): CTDirect traces the CTModels functions into ExaModels generators. CTParser then needs no :exa backend at all — @def emits only the :fun path, and the parser's job ends at building the CTModels model. The :exa code paths, PARSING_EXA, prefix machinery for Exa, get_build_examodel plumbing are deleted once parity is reached.
• Route 2 (expression IR, fallback): CTParser stores a neutral expression representation of the problem (dynamics/cost/constraint expressions, dimensions, names — it already manipulates exactly this) in the model, and CTDirect generates the Exa transcription from that IR with its own scheme/layout knowledge. The parser keeps a small "describe the problem as data" emitter; no if scheme == … remains here.

Consequences for open issues

• #65 non-uniform grid: becomes CTDirect's time_grid handling (already supported on the ADNLP path) — nothing to implement in CTParser; close it against the repatriation.
• #161 endpoint-control constraints (u_N = u_{N-1} for euler/midpoint): scheme knowledge → CTDirect's recipe (final-control trait, CTDirect §2.2/§3). If a short-term fix is wanted before the repatriation, implement it in the generated code behind the existing scheme branches, but mark it temporary.
• #134 exa multipliers for boundary/path constraints: solved by CTDirect §5.4/§9.2 (CTDirect owns the layout, so it can collect the multipliers) — keep the issue open here only as a tracking pointer.

Transition discipline

The current :exa backend stays frozen (bug fixes only, no new features) and serves as the comparison oracle for CTDirect's prototypes: same NLP dimensions, same solutions on the shared problem suite. It is deleted (route 1) or reduced to the IR emitter (route 2) once CTDirect's golden tests pass on both paths.

---

3. "1-D = scalar" simplification

Reference: Handbook dimension-and-shape.md. Status: to align; the parser is the emitter of user-facing code, so its compliance shapes everyone downstream.

Per the Handbook, the parser may emit [1]-indexed code (works with scalars) or scalar code; either honours the convention — what it must not do is force 1-vectors on the callers or wrap scalars into vectors.

1. Audit the :fun codegen for 1-D state/control/variable: emitted dynamics/cost/constraint closures should work when callers (CTFlows, CTDirect after its §7) pass scalars — [1]-indexed emission is the safe common denominator; drop any x → [x] vectorization at emission boundaries.
2. Simplify the dimension plumbing: places where the parser special-cases dim == 1 with wrapping/unwrapping should collapse to the single only/identity dimension-driven rule; remove now-useless scalar/vector duplication in utils.jl substitution helpers where applicable.
3. Contract tests (Handbook style): parse 1-D and n-D problems, call the generated functions with scalars (1-D) and vectors (n-D), assert both work and give equal results vs the length-1-vector form.
4. Coordinate with CTModels (storage passes functions verbatim) and CTDirect §7 (its callers must pass scalars) so the whole chain flips consistently.

---

4. Remove prefixes

Issue: #224. Status: decided in principle; small, high-leverage cleanup.

Generated code currently reaches CTBase/CTModels/ExaModels through mutable prefix Refs (PREFIX_FUN, PREFIX_EXA, prefix_fun!, prefix_exa!) so that the caller (OptimalControl) can point them at re-exported modules. Instead:

1. @reexport (or plain re-export) the needed module names from OptimalControl.jl, and prefix the generated code directly with CTModels.… / CTBase.… / ExaModels.… ab initio.
2. Delete the prefix Refs and setters — removes global mutable state, makes generated code readable and greppable, and kills the test_prefix*.jl special suite.
3. Note the interaction with §2: if route 1 wins, the Exa prefix disappears with the backend anyway; do the :fun prefix removal regardless.

---

5. Parsing robustness — bugs, strictness, slices

Issues: #197, #193, #139, #260 (finalize, with PR #251).

5.1 Bug sweep (#197)

Work through the catalogued items with a test per fix:

• @match patterns using bare control_fun instead of :control_fun — fix everywhere, add a regression test that would have caught the mis-route.
• as_range returning [x] where quoted code expects a range expression — introduce as_range_expr(x) = is_range(x) ? x : :($x:$x) and use it consistently in codegen paths.
• __wrap scoping hazard (local ex + throw(ex)) — tighten the generated error-handling block.
• Sweep the remaining [Dev] Code improvement and bug fixes #197 list (it contains further suggestions) and either fix or explicitly reject each with a comment in the issue.

5.2 Stricter parsing (#193, #139)

Silent mis-parses must become parse-time errors (ParsingError with position and hint):

• After replace_call, any leftover state/control symbol in an expression is necessarily a typo → error (not a gensym that fails later at runtime; the issue suggests gensym as the weaker fallback).
• Wrong call arguments like x(t0) when t0 is not the declared time boundary ([Dev] Improve parsing #139): detect at parse time that the argument of a state/control evaluation is neither the declared t, t0, tf nor a literal within the time range semantics — error with "did you mean x(0)?"-style hints. (The issue notes the tricky case where t0 exists in the eval scope — hence a parse-level rule on declared symbols, not an eval-level one.)
• General principle: every line either matches a known production or errors loudly — no expression silently swallowed.

5.3 Slice notation — finalize #260

PR #251 added comprehensive slice tests with known-broken cases. Finish the feature:

• Support x[1:3](t), x[1:end](t), x[1:n](t) (with n the declared dimension) uniformly in constraints, dynamics (∂(x[1:3])(t)), and costs — both backends.
• The end keyword and symbolic bounds require resolving against declared dimensions at parse time — the parser knows them; lower end/n to literals during normalization.
• Un-break the marked @test_broken cases; keep the test matrix from Add comprehensive slice notation tests #251 as the acceptance criterion.

---

6. Syntax extensions

Grouped because they are independent, user-visible, and each needs an explicit go/no-go with JB.

6.1 Coordinate-derivative aliases (#71)

Accept ẋ₁(t) == … / ẋ1(t) == … as sugar for ∂(x₁)(t) == …. Pure front-end: recognize the combining-dot-above character on state (component) names during normalization (Unicode dep already present). Small; needs a rule for what ẋ means when x is a vector (whole-state dynamics) vs a component alias.

6.2 Tensors / matrix-valued state and control (#181)

Go from x[i, j] (component × grid) to x[i, ii, (k,) j] — matrix (or complex-matrix) valued states, matrix-valued controls (extdet examples). This is not a parser-only feature: CTModels (storage, dimensions), CTDirect (layout flattening, sparsity), CTFlows (coercion) must all agree on the representation.

• Deliverable here: a cross-package design note first — how a matrix state is declared (x ∈ M(n, m), ℂ case), how it is stored (flattened with a shape trait? — relates to the Handbook dimension-and-shape philosophy), what the generated functions receive (a reshaped view? scalars stay scalars).
• Parser work (declaration syntax, index normalization) only after that note; do not ship a parser-side flattening convention that the rest of the ecosystem then has to reverse-engineer.

6.3 Incremental @def! (#32)

Build a problem incrementally, interpolating an existing one (@def! $ocp begin … end, à la @benchmark). Useful for parametric studies and continuation. Design questions: what is re-parsed vs mutated (CTModels models are immutable — so this is "parse extra clauses, rebuild"), interaction with initial_guess, and whether @def gains $-interpolation of scalar parameters as a first step (cheaper, frequently requested). Prototype the $-interpolation first.

---

7. Parsing engine redesign — staged pipeline over a typed IR

Reference: Handbook modules.md, types-traits-interfaces.md. Status: design proposal, not started. Best done after (or jointly with) the §2 route decision, since the IR is exactly the "neutral expression representation" of route 2.

The core problem of the current engine: the "one pass" does everything at once — syntax recognition, semantic analysis and code generation are interleaved inside each @match arm and duplicated per backend. Robustness bugs (§5) and extension cost both stem from this.

Proposal: a staged compiler-style pipeline where each @def line is parsed into a typed IR node (data, not code), validated, and only then emitted:

# Stage 0 — surface normalization (pure Expr → Expr, no semantics)
#   ẋ₁(t) → ∂(x₁)(t) (§6.1), unicode aliases, slice sugar

# Stage 1 — Grammar: each line → one IR node (data)
abstract type AbstractNode end
struct StateDecl  <: AbstractNode; name::Symbol; dim; components; line::LineNumberNode end
struct Dynamics   <: AbstractNode; coord; rhs::Expr; line::LineNumberNode end
struct Constraint <: AbstractNode; lb; body::Expr; ub; label; line::LineNumberNode end
# parse_line(ex, line)::AbstractNode — MLStyle @match INSIDE each rule, returning data

# Stage 2 — Semantics: symbol table + validation on the IR
#   dimensions, constraint classification (boundary/path/box/mixed),
#   end/n range resolution (§5.3), leftover x/u symbols (#193), x(t0) checks (#139)
validate!(table::SymbolTable, node::AbstractNode)          # multiple dispatch

# Stage 3 — Emission: backends consume the VALIDATED IR
emit(::FunTarget, node::AbstractNode, table)::Expr         # per-node, per-target methods

Why more robust

• Parse, don't validate: a line matching no production fails at stage 1 with its LineNumberNode; nothing is silently swallowed. The strictness rules of §5.2 become centralized stage-2 checks on data instead of guards scattered through codegen.
• Error accumulation: stages are separable, so one @def can report all its errors with positions in one shot, instead of dying on the first.
• Fine-grained testing (Handbook testing.md, feeds §8): grammar tests (surface → IR, no codegen), semantic tests (hand-built IR → expected errors), emission tests (IR → code → behavior) — each stage a pure, unit-testable function.

Why still simple to extend

• Adding a production = adding one file: node struct + parse rule + validate! method + emit methods — plain open multiple dispatch (OCP), no edit to a central @match. MLStyle stays, inside each rule, for what it is good at.
• Adding a backend = adding emit methods, not duplicating the grammar.
• No parser-combinator framework needed: the DSL is line-oriented; an ordered production table (the PARSING_DIR idea, at production granularity, returning data) suffices.

Synergies

1. The IR is route 2's neutral representation (§2 / CTDirect §5): if function tracing falls short, the validated IR is what CTParser stores in the model for CTDirect to transcribe. Engine redesign and Exa repatriation converge on the same object — design them together.
2. Incremental @def! (Incremental @def! #32) becomes IR merging (re-validate, re-emit), not surgery on generated code.
3. Slices / end ([Bug] Comprehensive Slice Notation Tests #260, §5.3) resolve at stage 2 where dimensions are known.
4. Pretty-printing the IR back to canonical @def syntax gives "did you mean" error hints, an auto-derivable grammar reference (§9.2), and doctests that cannot drift.
5. Safe migration: build the pipeline greenfield next to the current one-pass, with the existing test suite as oracle (same strategy as CTDirect's ground rule); switch when green, delete the one-pass.

Submodule split (follows the stages)

Submodule	Responsibility	Absorbs
IR	node types, symbol table, pretty-printer	new
Grammar	stage 0–1: normalization, production table, parse_line rules	pattern-matching part of onepass.jl, parts of utils.jl
Semantics	stage 2: validate! methods, strictness (§5.2), error accumulation/reporting	ParsingInfo (p), scattered checks
FunBackend	stage 3 for CTModels codegen	:fun half of onepass.jl
ExaBackend (transitional)	frozen :exa codegen, then deletion (route 1) or replaced by the IR handoff itself (route 2)	:exa half of onepass.jl
InitialGuess	@init parsing (same pipeline, own productions)	initial_guess.jl
Defaults	defaults & option plumbing	defaults.jl

Handbook rules apply: one directory + same-named manifest per submodule, top-level CTParser.jl exports nothing, qualified imports (using MLStyle: MLStyle — currently bare using MLStyle, using Parameters, using OrderedCollections all pollute scope), naming conventions symbol / _helper / __default, dependency DAG (IR → Grammar → Semantics → backends).

Also fold in: move the header TODO comments of onepass.jl into tracked issues; kill the *_bis.jl file-duplication pattern (belongs to §8).

Deliverable order: (1) short design note fixing the IR node set and stage contracts, written together with the §2 route decision; (2) IR + Grammar for the declaration productions only, oracle-tested against the current parser; (3) semantics + strictness rules (§5.2 lands here, on the new engine); (4) FunBackend emission, full-suite parity; (5) switch + delete one-pass.

---

8. Test suite restructuring

Reference: Handbook testing.md. Status: not compliant (flat scripts, _bis duplicates, shared utils.jl).

1. Adopt the file template: module-wrapped files, qualified imports, test_<name>() entry redefined at outer scope, top-level fakes, VERBOSE/SHOWTIMING via Main.TestOptions.
2. Organize by functionality: suite/grammar/ (productions, normalization, aliases, slices — the Add comprehensive slice notation tests #251 matrix lives here), suite/strictness/ (error paths of §5.2 — every ParsingError with @test_throws and message checks), suite/codegen_fun/ (generated CTModels code: build + call the functions, scalar/vector 1-D equivalence of §3), suite/codegen_exa/ (frozen oracle suite while §2 transitions), suite/initial_guess/, suite/integration/ (end-to-end @def → solve smoke tests, shared problems with CTDirect).
3. Golden oracle role: the :exa tests double as the comparison baseline for CTDirect's repatriation (§2) — keep them green and pinned until the switch.
4. Merge the *_bis.jl shadow suites into their primaries; remove order dependence; keep coverage.jl tooling.

---

9. Documentation

Reference: Handbook VITEPRESS-DOC.md. Status: minimal (single index.md).

1. Vitepress migration (mechanical, per the Handbook recipe — deps, make.jl, generate_template, config.mts patches).
2. DSL reference: a complete grammar page — every accepted production (time, state, control, variable, aliases, constraints forms, dynamics incl. coordinate/slice forms, costs), with the strictness rules of §5.2 and the error messages users will see. Today this knowledge only exists in onepass.jl pattern matches.
3. Guides: initial guess (@init), backend story after §2 (what :exa means for users — ideally: nothing, it's a solver-side choice), migration notes for removed prefixes (§4) and any syntax changes (§6).
4. Doctests for the DSL examples so the grammar page cannot drift from the parser.

---

Priority summary

Priority 0:

• §1 ExaModels 0.11 migration — compat blocker for the ecosystem; do it on the current code, before any restructuring.

High priority:

• §2 discretization out of the parser — joint design with CTDirect §5 (feasibility prototype early); freeze :exa meanwhile.
• §7 engine design note (IR + stages) — written together with the §2 route decision (the IR is route 2's neutral representation); implementation follows the §7 deliverable order.
• §5.1 bug sweep ([Dev] Code improvement and bug fixes #197) — correctness fixes on the current engine (it stays in service until §7 ships).
• §4 prefix removal ([Dev] Remove prefixes #224) — small, removes global state; coordinate the OptimalControl re-export.

Medium priority:

• §7 pipeline implementation (IR + Grammar + Semantics + FunBackend, oracle-tested) — §5.2 strictness and §5.3 slices land on the new engine rather than being retrofitted onto the one-pass.
• §3 "1-D = scalar" simplification — synchronized with CTDirect §7 / CTModels; cheapest done in the new FunBackend emission.
• §8 test restructuring — progressive; the :exa oracle suite first (it guards §2 and §7).
• §6.1 derivative aliases ([Dev] Coordinate derivatives aliases #71) and §6.3 $-interpolation prototype (Incremental @def! #32) — small syntax wins (#71 is a stage-0 normalization rule in the new engine).

Lower priority:

• §6.2 tensors/matrices ([Dev] Adding tensors #181) — cross-package design note first; implementation follows the ecosystem decision (and lands as new IR nodes, §7).
• §9 documentation — mechanics anytime; the DSL grammar reference derives from the §7 IR/productions.

---

Dependency graph

§1 ExaModels 0.11 (on current code) ──→ everything Exa-related after it

CTDirect §5 design note (tracing vs IR) ──┬─→ §2 discretization out of the parser
                                          │      └─→ closes #65, #161, #134 (moved to CTDirect)
                                          └─→ §7 engine design note (IR = route 2's representation)
                                                 └─→ §7 pipeline (greenfield, one-pass as oracle)
                                                        ├─→ §5.2 strictness, §5.3 slices (on the new engine)
                                                        ├─→ §3 1-D = scalar emission
                                                        └─→ §6.1/§6.2/§6.3 syntax (stage-0 rules / new IR nodes)

§5.1 bugs / §4 prefixes — independent, on the current engine, can start now
§6.2 tensors — cross-package design note (CTModels/CTDirect/CTFlows) before any parser code
§8 tests — :exa oracle suite first (guards §2 and §7), then tracks each section
§9 docs — DSL grammar reference derives from §7 IR; Vitepress mechanics anytime

---

Coordination points

• CTDirect: §2 is the mirror of CTDirect roadmap §5 (routes, oracle, deletion criteria); §3 with CTDirect §7 (scalar callers); [Dev] Constrain endpoint controls #161/[Dev] Add non uniform grid for exa #65/[Dev] Exa retrieve multipliers for boundary and path constraints #134 transfer there.
• CTModels: neutral expression IR (if route 2), tensor/matrix representation (§6.2), verbatim function storage under the scalar convention (§3).
• CTFlows: coercion side of §3 and §6.2 (already "1-D = scalar" compliant — reuse its vocabulary).
• OptimalControl: re-exports for prefix removal (§4), compat bumps (§1), user-facing deprecations (§6, §2).