Define left recursion by the left-corner relation; reject indirect/hidden cycles at build time

[johnsoncodehk]

Problem

The parser's termination guarantee rested on a local syntactic check — isLeftRecursive, which asked of each rule in isolation: does some alternative's items[0] reference the rule itself? That is an under-approximation of left recursion. It recognizes only the length-1, non-nullable witness and misses:

• Indirect cycles — A → B | …, B → A | …
• Recursion hidden behind a nullable prefix — A → opt(x) A … (the first item is opt(x), not a bare self-ref)

Both still re-enter the rule at the same input position, so recursive descent recurses without consuming and overflows the stack. The failure mode was the worst kind: nothing flagged the grammar at build time (defineGrammar/createParser both succeed), and parsing certain inputs threw an uncaught RangeError: Maximum call stack size exceeded from deep inside the engine, naming no offending rule.

Minimal repro (a single-token input is enough):

A → B | W ;  B → A | W      // indirect
A → opt(D) A W | W          // nullable-hidden

Both RangeError on input "a", while the equivalent direct form A → A W | W parses fine — proving the gap is specifically the syntactic detector, not left recursion as such.

Root cause

Termination depended on a single static, local, syntactic classifier that under-approximates a global, nullable-aware property — whether a rule lies on a cycle in the left-corner relation — with no runtime backstop. The same engine already computes FIRST and nullability as global fixpoints over the rule graph (cycle-safe by construction); left-recursion detection alone reached for a syntactic shortcut, which is exact only for the direct operator/postfix recursion the shipping grammars happen to use.

Fix

Make the left-corner relation the single source of truth:

• isLeftRecursive(rule) is now "the rule reaches itself through the transitive closure of the left-corner edge map" — nullable-aware (reuses nullableRules/exprNullable); op/prefix/postfix are left-edge barriers (they consume an operator token). This catches direct, indirect, and nullable-hidden recursion uniformly.
• items[0] === self is demoted from "the definition of left recursion" to merely which alternatives the local atom/continuation (and Pratt NUD/LED) transform can peel into an iterative loop (peelsDirect).
• A left-recursive rule is handleable iff peeling its direct self-alts breaks every cycle (its residual graph is acyclic) → classifyLeftRec / Pratt handle it unchanged. Otherwise it is rejected at build time with a diagnostic naming the residual cycle (A → B → A). The engine deliberately does not parse indirect/hidden left recursion — for a grammar→highlighter generator that is an authoring error, and rejecting it early (like tree-sitter/yacc reporting conflicts) is the correct behavior.

prattRules/hasMarker, classifyAlts's peeling, and the entire Pratt parse path are untouched.

Behavior identity

This is a structural unification, not a capability change:

• All 7 grammars regenerate byte-for-byte identically — regenerating leaves every *.tmLanguage.json / *.monarch.json / *.cst-types.ts / tree-sitter/** unchanged.
• npm test (sanity) unchanged: 15/15.
• New test/left-recursion.ts (npm run test:left-recursion): direct LR parses; indirect and nullable-hidden LR are rejected at build time (the test asserts rejected, not overflow, so it guards the regression); the diagnostic names the rule and cycle. 4/4.