fix(repl): normalizer PREV_BINOP class for binop-at-EOL continuations (#620 burndown)

[jsavin]

Summary

5th hole in normalize_newlines_to_semicolons (after #628/#635/#640): a UserTalk script ending a line with a binary-operator token (and, or, ==, >=, <=, !=, >, <, +, -, *, /, ,, () had a synthetic ; inserted before the continuation, producing source like x and; y that the parser rejected. The 5 failing sys.* integration tests using (typeof(...) == stringType) and\n (typeof(...) == stringType) patterns were the visible symptom; the same hole catches any multi-line expression with a trailing binop.

Add a PREV_BINOP row to the transition table that suppresses ; for every next-kind — a continuation is a continuation regardless of what follows.

Implementation

• New PREV_BINOP member in prev_kind_t. The transition table grows one row; all five cells are suppress.
• New prev_is_binop(out, dst) helper walks back through the output buffer to detect the trailing operator token. Single-char operators (+ - * / , ( > <) match on the last byte. Multi-char operators require disambiguation:
  • = is BINOP only if the byte before is =, >, <, or ! (so == / >= / <= / != match but bare assignment = does not).
  • and / or require a word boundary before the keyword (so identifiers like band or door are not misclassified).
• classify_prev_full(prev_nonws, out, dst) is the buffer-aware wrapper used by the normalizer loop. Falls back to single-byte classify_prev for anchor bytes (\0 ; { }), then upgrades PREV_OTHER to PREV_BINOP when prev_is_binop matches.
• Cells changed in kEmitSep: new BINOP row, all-zero. Existing rows untouched.

Judgement calls

• Unary - is over-suppressed. x = -\n 1 (unary minus split across lines) is statically indistinguishable from binary - without a real tokenizer. We treat all trailing - as BINOP. The worst case is a script that was already malformed losing one parse-error surfacing, which is preferable to silently mis-normalizing the common x +\n y and x -\n y patterns.
• ) is NOT BINOP. Close paren is end-of-expression, not continuation. Same for } and ; (handled by their own prev-kind classes).
• No appisrunning work. The bundled task described a missing headless sys.appisrunning implementation, but tests/headless_sys_verbs.c already implements it via pgrep -x (committed earlier, marked COMPLETE). Manual testing confirms sys.appisrunning("frontier-cli") returns true when called from inside frontier-cli. The 5 failing tests are 100 percent normalizer-caused (all use and\n continuations). Change B from the task was unnecessary.

Tests

8 new tests in protocol_eval_compile_errors.yaml:

• 6 positive-path BINOP continuations (boolean and / or, relational ==, arithmetic +, comma, open paren)
• 1 word-boundary regression guard (local (band = 7)\n return band — identifier ending in and must still get ;)
• 1 disambiguation regression guard (x =\n 5 — single = is assignment, must still get ;; result is a parse error, locked in as success: false)

Test plan

• make -C frontier-cli -j$(sysctl -n hw.ncpu) — clean build, no new warnings
• ./tools/run_headless_tests.sh — 489/489 pass
• cd tests && make test-integration — 70 to 65 failures (-5 sys cluster), +8 new tests all green. Net: 2032 to 2045 passing, 2309 to 2317 total
• All 39 pre-existing tests in protocol_eval_compile_errors.yaml still pass unchanged
• Manual smoke: sys.appisrunning("frontier-cli") returns true via --protocol script/eval — appisrunning was already working

Per-cluster count change

• sys verbs - system information suite — fixed (used and\n)
• sys verbs - all info verbs return strings — fixed (used and\n)
• sys verbs - numeric verbs return long — fixed (used and\n)
• sys verbs - boolean verbs return bool — fixed (used and\n)
• sys environment verbs - isolation test — fixed (used and\n)

Generated with Claude Code (https://claude.com/claude-code)

[jsavin]

Local /gate round 2 (post-P1 fixes)

Verdict: PASS — both P1 findings addressed.

Round 1 → Round 2

Round 1 (35290de40) findings:

• ✅ P1 Repo Modernization: 64‑bit/ARM + DB Header Rev; Headless/Portable Runtime; Headless EFP Routing for file.* and Tests #1 (test gaps for >=, !=, *): 3 new tests added pinning these byte-classes.
• ✅ P1 Fix headless runtime warnings #2 ('or' word-boundary): 1 new test (door variable) symmetric to the band test for 'and'.
• ⏭️ P2 docs: UTF-8 transition plan #3 (100-col table-row): pre-existing pattern, alignment is the documentation — left as-is per bar-raiser's recommendation.
• ⏭️ P2 planning: relocate UTF-8 plan and add phase directories #4 (for/while at EOL): not a regression; tracked as the next hole if anyone hits it. No change in this PR.

Tests at HEAD fa2a8ebeb

• Unit: 489/489
• Integration: 2321 / 2050 passed / 64 failed / 207 skipped (+4 total tests, all 4 new BINOP regression tests green)
• protocol_eval_compile_errors.yaml: target file 47/47 pass

Why not a full multi-reviewer re-gate

Round-2 diff is tests-only (4 new YAML entries, +48 lines). No code changes to repl_variables.c. Re-running bar-raiser/security/concurrency would consume reviewer time for zero new signal on the C diff. Round-1 review of the underlying code stands.

Ready to merge.

[jsavin]

/auto Summary

What went well:

• The /gate review caught the bug-in-the-bug-in-the-bug again. After fix(repl): preserve try/else and if/else across newlines in protocol mode #635 and fix(repl): suppress spurious ';' on trailing newline / trailing ';' (#620 burndown) #640 had each found subtle holes the prior round missed, fix(repl): normalizer PREV_BINOP class for binop-at-EOL continuations (#620 burndown) #643's bar-raiser found that the test coverage for the new BINOP byte-classes was incomplete — >=, !=, *, and the symmetric or word-boundary were unverified. Round 2 added 4 tests to close the gap.
• The triage path leading to this PR was high-leverage. What looked like "5 flaky sys tests" turned out to be a deterministic compiler bug affecting any UserTalk script with operator-at-EOL continuation. The fix is in the right layer (normalizer) and the test matrix now pins every documented BINOP byte-class.
• The bundled task spec was correctly rejected by the agent — sys.appisrunning was already implemented; bundling a redundant fix would have added noise. Good judgment.

What could improve:

• Anthropic API hit a 529 storm during the gate dispatch (4 reviewers all 529'd in their first attempt, then succeeded on retry). Not Frontier-specific. If we hit this again during /auto chains, the right move is to wait a few minutes and retry rather than dispatching new parallel agents.
• Edit/Write tools silently no-op'd on frontier-cli/repl_variables.c for the PR agent. They worked around via Python read/replace/write. Worth filing if reproducible — this is the second time the same agent worktree showed file-edit drift (also seen in the security review of fix(repl): normalizer PREV_BINOP class for binop-at-EOL continuations (#620 burndown) #643 round 1).

Recommendations:

• The "for / while / if at EOL" cell is the next normalizer hole. Not regressed by this PR (pre-existing). File a follow-up if anyone hits it; otherwise it can sit until a real script trips it.

🤖 Generated with Claude Code /auto