ltm: C-LEARN LTM compile-time dominated by per-fragment fixed overhead (DimensionsContext clone, O(D^2) intern_name, repeated parse)

[bpowers]

Context

After the 2026-05-31 LTM compile-time optimization work (commit bfff92e8 "engine: fix O(N^2) blowups in LTM compilation", through the bfff92e8..51392894 cluster on main), LTM compilation of the C-LEARN v77 model dropped from ~47.6s to ~6s. Profiling the residual ~6s shows the remaining cost is dominated by per-fragment fixed overhead that is paid once per LTM synthetic variable + helper. C-LEARN mints ~37k of these, so even small per-fragment constants dominate.

The two hot fragment-compile entry points are in src/simlin-engine/src/db/ltm/compile.rs:

• compile_ltm_equation_fragment (around line 241; Module built at lines 934-935)
• compile_ltm_implicit_var_fragment (around line 1187; Module built at lines 1732-1733)

Findings

1. Per-fragment deep-clone of the DimensionsContext and converted dims

Each per-fragment compiler::Module construction does:

dimensions_ctx: (*dim_context).clone(),
dimensions: converted_dims.to_vec(),

(compile.rs:934-935 and compile.rs:1732-1733)

DimensionsContext is a HashMap of dimensions with per-dimension element maps; cloning it (plus the converted-dims Vec) per fragment deep-copies the entire project dimension set ~37k times for C-LEARN.

Fix direction: make DimensionsContext / Dimension internals Arc-backed so a clone is a refcount bump, or let Module borrow them rather than own a clone.

2. Compiler::new -> populate_dimension_metadata interns names with an O(D^2) linear scan, then discards the result

Compiler::new (src/simlin-engine/src/compiler/codegen.rs:49) calls populate_dimension_metadata (codegen.rs:87), which iterates all project dimensions per fragment and interns each dimension/element name via intern_name (codegen.rs:109). intern_name is a linear scan over the names vec:

if let Some(idx) = self.names.iter().position(|n| n == name) {
    return idx as NameId;
}

That is O(D^2) string comparisons per fragment compile, repeated ~37k times. The dimension tables produced this way are discarded for fragment compiles -- the assembly stage builds its own.

Fix direction: hash-based interning for intern_name, or skip dimension-metadata population entirely for per-variable fragment compilation (the produced tables are unused on this path).

3. Each LTM equation is parsed 2-3 times during assembly

The same LTM equation text is parsed repeatedly across:

• model_ltm_implicit_var_info
• compile_ltm_synthetic_fragment
• assemble_module's implicit-var loop (via parse_ltm_var_with_ids)

Fix direction: parse once and thread the parsed AST through, or memoize the parse keyed on the synthetic var.

4. Identifier canonicalization/interning under a global lock is ~15-20% of compile time

CanonicalStorage::intern (src/simlin-engine/src/common.rs:226) routes through the global sharded interner (Interner::global(), common.rs:72 -- an array of Mutex<Shard>). At ~37k fragments this lock traffic is ~15-20% of compile time.

This overlaps with #317 (intern Ident<Canonical> via salsa) -- #317 would reduce the general hashing/cloning cost, but the LTM fragment path's volume of intern calls is the specific amplifier here, and findings 1-3 above are LTM-fragment-specific and not covered by #317.

Why it matters

Maintainability and developer/agent experience: 6s to compile a single (large, but real) model with LTM enabled is slow enough to discourage iterative use of LTM on production-scale models. The remaining cost is fixed overhead multiplied by synthetic-var count, so it scales poorly as models grow. None of it is intrinsic work -- findings 1-3 are pure waste (deep-cloning immutable data, computing-then-discarding dimension tables, re-parsing identical text).

Components affected

• src/simlin-engine/src/db/ltm/compile.rs (fragment-compile entry points)
• src/simlin-engine/src/compiler/codegen.rs (Compiler::new, populate_dimension_metadata, intern_name)
• src/simlin-engine/src/common.rs (global interner; shared with Intern Ident<Canonical> strings via salsa::interned to reduce hashing and cloning overhead #317)
• DimensionsContext / Dimension definitions (for the Arc-backed-clone direction)

Relationship to existing issues

• Follow-up to the LTM compile-time work on 2026-05-31 (commit bfff92e8 and the bfff92e8..51392894 cluster).
• Part of the LTM epic epic: Loops that Matter (LTM) -- tracking known issues and improvements #488.
• Overlaps partially with Intern Ident<Canonical> strings via salsa::interned to reduce hashing and cloning overhead #317 (general Ident<Canonical> interning) for finding 4 only; findings 1-3 are LTM-fragment-specific and independent.
• Distinct from ltm-finding: SearchGraph clones Ident<Canonical> in per-timestep DFS hot path #481 (SearchGraph DFS clone in discovery/post-sim hot path) and engine: optimize the graphical-function Lookup hot path (~4% of the C-LEARN run) #602 (graphical-function lookup at simulation runtime), which are not compile-time.

Discovered

Identified during profiling of the 2026-05-31 LTM compile-time optimization work; deferred as the next layer of compile-time reduction after the O(N^2) blowups were removed.

[bpowers]

Data point: the GH #738 fix (dependency-aware lowering scope for LTM fragments, commit 1fe8b94) added per-fragment cost on top of the overheads tracked here, and the mitigation commit bd4376e recovers most but not all of it.

Interleaved release A/B on C-LEARN (warm builds, 4 rounds for compile / 3 for the pinned test):

metric	pre-#738	#738 unmitigated	#738 + mitigation (bd4376e)
cold LTM-enabled compile_project_incremental	5.99s	7.53s (+26%)	6.31s (+5.2%)
clearn_pinned_climate_loop_is_scored	9.25s	10.91s (+18%)	9.65s (+4.3%)

The mitigation eliminated the duplicated per-fragment classify_dependencies walk (the helper now returns the classification for reuse) and gates the scoped re-lower on a cheap syntactic reducer-call check, so reducer-free fragments pay nothing.

The ~5% residual is the per-reducer-fragment work itself: a second lower_variable pass plus per-dep VariableStage0 clones (parse_source_variable_with_module_context is salsa-cached, but the Stage0 variable is cloned into the mini ModelStage0) and an uncached parse_var per arrayed implicit-helper dep. C-LEARN is reducer-heavy, so many fragments take this path. This is the cost of compiling those fragments correctly -- before #738 they silently compiled to constant-0 stubs whenever the reducer argument was an array expression -- so it cannot be removed outright, only made cheaper. Plausible levers, all in this issue's per-fragment-overhead territory: salsa-caching the scoped lowering per (var, dep-set), sharing parsed Stage0 deps across fragments of the same model (arena or Arc instead of clone), and caching the implicit-helper parse_var results. Warm/incremental recompiles are unaffected (salsa); WRLD3 is unaffected.

[bpowers]

Another data point in this issue's per-fragment-overhead territory, from the GH #741 work (commit cd96ffcf on ltm-fix-batch-2, flagged during adversarial review): the new implicit-helper diagnostics leg double-compiles every LTM implicit helper, uncached.

model_ltm_fragment_diagnostics (src/simlin-engine/src/db/ltm/compile.rs:1568) now probes each implicit helper for silent compile failure by calling compile_ltm_implicit_var_fragment (compile.rs:1632 → 1678). That function is a plain pub(crate) fn, not salsa-tracked, so the probe pays a second full helper compile on top of the one assemble_module already does for the same revision. The function's own perf comment (compile.rs:~2214) notes it runs once per LTM implicit variable — ~145k times on C-LEARN — so the diagnostics pass roughly doubles that call count whenever it runs. The compile_var_fragment precedent the pass cites IS salsa-tracked (#[salsa::tracked(returns(ref))], src/simlin-engine/src/db/fragment_compile.rs:25), so the analogous probe on the non-LTM path is a cache hit; the LTM probe is not.

Magnitude is unmeasured and the blast radius is bounded: the pass only runs on diagnostics collection (model_all_diagnostics with LTM enabled), not on every compile. But LTM-heavy get_errors flows pay it — libsimlin simlin_project_get_errors under the #466 LtmEnabledGuard, and MCP read_model/edit_model with #662.

Fix direction: make compile_ltm_implicit_var_fragment salsa-tracked, mirroring compile_var_fragment. Note the diagnostics probe deliberately uses the empty module-input set while assembly uses the instance's inputs — for the root model those are byte-identical, so salsa would dedupe the dominant case; sub-model instances with inputs would still compile twice unless the probe is keyed differently. The signature currently takes borrowed args (&LtmImplicitVarMeta, &ModelDepGraphResult, &[String]), so tracking it means moving those to salsa-interned/tracked inputs; alternatively, memoize probe results from assembly and have the diagnostics pass read those.