kernel: scanner only recognizes MacRoman «» (0xC7/0xC8), not UTF-8 (0xC2 0xAB/0xC2 0xBB)

[jsavin]

Context

UserTalk uses «…» (U+00AB / U+00BB) as a comment delimiter, alongside // for line comments. The convention dates back to the MacRoman era where « was a single byte 0xC7 and » was 0xC8.

The kernel scanner (Common/source/langscan.c) currently only recognizes the MacRoman single-byte encoding. Protocol-mode script/eval receives scripts as UTF-8 (via JSON), where « encodes as the two-byte sequence 0xC2 0xAB and » as 0xC2 0xBB. The scanner doesn't recognize these bytes as comment markers and treats them as data — producing compile errors or, worse, scanning into the comment body as code.

Surfaces affected

• Protocol script/eval (CLI --protocol, integration test runner) — every script body arrives as UTF-8 from cJSON. «…» comments don't work.
• .ut file import if any path ever decodes them as UTF-8 (currently .ut files are stored/read as MacRoman, so this is fine today, but the divergence is fragile).
• Anything that round-trips UserTalk through a UTF-8 layer (a future filesystem-canonical pipeline per ADR-017, for example).

What works today

• Legacy .ut files written in MacRoman — fine.
• // line comments — work everywhere.
• The protocol-mode newline normalizer (normalize_newlines_to_semicolons in frontier-cli/repl_variables.c) correctly identifies UTF-8 0xC2 0xAB/0xC2 0xBB as curly comments for its lookahead (per fix(repl): preserve try/else and if/else across newlines in protocol mode #635) — so the normalization layer is internally consistent. This issue is purely a kernel-scanner gap.

What to do

Extend scanner comment-start detection in langscan.c to accept the UTF-8 two-byte encoding alongside the MacRoman single-byte encoding:

• 0xC2 0xAB (UTF-8 «) — equivalent to 0xC7 (MacRoman «)
• 0xC2 0xBB (UTF-8 ») — equivalent to 0xC8 (MacRoman »)

Single source of truth would be best — the scanner should treat both forms identically when entering/exiting a curly comment.

Acceptance criteria

• A protocol script/eval with a «…» curly comment compiles cleanly and produces the expected return value
• An integration test in protocol_eval_compile_errors.yaml exercises «…» comments in protocol mode (single-line, multi-line, between } and else)
• MacRoman .ut files continue to work unchanged
• Unit and integration suites both green

Discovered

Sweep done in PR #636 — the normalizer-side comment skipping was verified correct, exposing this kernel-side gap. The current behavior was documented with a skip:d test that codifies the limitation.

Refs: #586 (UserTalk parser quirks), #635 (normalizer lookahead), PR #636 (regression sweep).