Restrict JSON inter-token whitespace to RFC 8259's ws

[ESRogs]

Overview

The JSON parser accepts whitespace between tokens that RFC 8259 does not allow. Inter-token skipping uses Str.trim_start, whose Unicode whitespace set is wider than the RFC's ws (space, tab, \n, \r) — so documents containing, e.g., a form feed or a no-break space between tokens parse successfully. This PR makes token separation exactly the RFC's four bytes. (This is the open question flagged in roc-lang#10145; it also resolves an inconsistency noted there — the scalar splitter already recognized only the RFC bytes, so non-RFC whitespace was rejected after a scalar but accepted everywhere else.)

Detail

Inter-token skipping now goes through a JSON-specific json_trim_start: it locates the first non-whitespace byte with List.find_first_index over the RFC whitespace set and drops the prefix as a zero-copy slice. Only the whitespace prefix itself is ever revalidated, so a call costs O(leading whitespace). All 65 skip sites in the parser switch to it; the tokenizer's delimiter set is unchanged (it was already RFC-only).

Tests: the four RFC whitespace bytes are verified skippable around tokens; a form feed (0x0C), a no-break space (U+00A0), and a vertical tab (0x0B) between tokens each reject the document; and a no-break space is rejected at each grammar position separately (before a value, after a key, before and after a colon, between fields, before a closer, after an array opener, inside a skipped object, and trailing after the document).

Why rejection works

json_trim_start never rejects anything itself: it trims leading RFC whitespace and returns everything else untouched, so a non-RFC whitespace character between tokens is simply left in place. Rejection then depends on an invariant of the parser: every trim site is immediately followed by a strict expectation — one of

• an end-of-input check (Str.is_empty after trimming the tail),
• a match against a specific structural character (:, ,, {, }, [, ], "), or
• a dispatch over the value-start set (", a digit, -, t, f, n, {, [).

No byte of any non-RFC whitespace character can satisfy any of these: the ASCII controls (0x0B, 0x0C) are in none of the sets, and every multi-byte whitespace character begins with a UTF-8 lead byte (0xC2, 0xE1, 0xE2, 0xE3), which is in none of the sets either. So the leftover character fails the very next match and the document is rejected.

This means the guarantee is only as strong as the "strict expectation" invariant across all 65 call sites — a site that matched leniently after trimming (for example, a Str.drop_prefix that no-ops on mismatch and continues down a tolerant path) would silently accept the document instead. The call sites have been audited for exactly that pattern, and the test suite pins the behavior at each grammar position.

Performance

Interleaved against the base branch (median wall time for a loop of 20,000 Json.parse calls, default LLVM --opt=speed, macOS arm64): 500 short string fields 1.40s → 1.75s (+25%), 500 numeric fields 1.54s → 1.85s (+20%).

The whitespace skip runs at every token boundary, and List.find_first_index with a predicate closure carries a per-call cost the compiler does not yet eliminate. Measured decomposition on the same workload: roughly a third of the overhead is loop entry (materializing the iterator and stepping its protocol for a typical 0–3 byte whitespace run), and the rest is the function boundary, predicate closure, and Try result plumbing. A hand-written while loop over the bytes with unchecked indexing measures +3–6% against the same base.

The find_first_index form is used deliberately: it states the intent directly, and it is the style the compiler's iterator work is designed to make free — per-chain iterator representations landed in roc-lang#10064, and compiler-side inlining ahead of LLVM is planned. If parsing throughput matters before that lands, the while-loop form of this helper is a drop-in replacement, and a native ASCII-set trim intrinsic remains an option beyond that.