Skip to content

emit-portable: segment reuse for newline-interleaved entries (#57 S8) - #70

Merged
johnsoncodehk merged 1 commit into
masterfrom
portable-seg-reuse
Jul 12, 2026
Merged

emit-portable: segment reuse for newline-interleaved entries (#57 S8)#70
johnsoncodehk merged 1 commit into
masterfrom
portable-seg-reuse

Conversation

@johnsoncodehk

Copy link
Copy Markdown
Owner

Extends tree reuse (#69) to newline-interleaved entry grammars — the env-spec/dotenv family that is monogram's core use case. Previously these grammars only got the lexer-side incremental path (windowed relexing) and always reported reused: 0.

Shape B eligibility

Reuse eligibility now also accepts entries of the form [opt(rule)] star(seq(tok, opt(rule))) (envspec's Program = [opt(Stmt), many(Newline, opt(Stmt))]). The reuse unit becomes a segment — one star iteration (Newline leaf + optional Stmt), or the optional head statement. Segments are bookkeeping only: the CST keeps its flat [Stmt?, Newline, Stmt?, …] children and the JSON output is byte-identical to master.

The reuse decision is the same algorithm as #69, applied to segments: keep-prefix by lookahead watermark (ext <= prefix), keep-suffix by tokStart >= oldN - suffix, exact boundary hit for adoption, zero mutation on every fallback path. Two envspec-specific hazards are pinned by mutation tests:

  • a segment's ext must include the failed opt(Stmt) probe after the Newline (span end alone is unsound);
  • zero-width Newline tokens share byte offsets with adjacent statements, so all decisions use token indices; bytes are only used for the adoption shift.

Shape A grammars (calc/javascript/…) are untouched — gate numbers and CST bytes are identical to master.

Go lookahead watermark fix (review round)

Cross-target fuzzing during review caught reused diverging between targets (Go one segment higher than TS/Rust) on comment/blank-line documents. Root cause: Go's matchLit/matchTok read toks[pos] directly without updating maxLook, while TS/Rust route probes through peek(). A failed probe is lookahead — missing it lets keep-prefix admit a segment whose parse examined the damaged region. Benign in the caught instance (treeEq stayed true) but unsound in general; Go matchers now update the watermark, and the three targets agree everywhere the fuzz can reach.

Validation

  • Gate: envspec edit scenarios extended (large-doc mid edit, head/tail edits, blank-line folding, flow-suspension interior, multi-batch), all run per target in validated + fast modes; edit ≡ fresh byte-compared; shape A scenario numbers asserted unchanged.
  • Review fuzzing: 40 random envspec multi-batch sessions × 3 targets (exit codes, CST bytes, align fields cross-target identical; treeEq/streamEq true on accepts), plus re-runs of the emit-portable: incremental tree reuse for portable targets (#57 S7a–S7c) #69 calc/javascript fuzzes — all green after the Go fix.
  • 3 executor mutation tests (probe-lookahead ext, byte-coordinate boundary, unconditional head keep) all caught by treeEq.
  • Full verify: tsc, emit-parity 5 gates, newline regression, 6 corpus adversarial scripts — ALL PASS.

Co-authored-by: Cursor <cursoragent@cursor.com>
@johnsoncodehk
johnsoncodehk merged commit 9260401 into master Jul 12, 2026
3 checks passed
@johnsoncodehk
johnsoncodehk deleted the portable-seg-reuse branch July 12, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant