Add Str.split_at_utf8_byte#1
Closed
ESRogs wants to merge 5 commits into
Closed
Conversation
ESRogs
force-pushed
the
split-at-utf8-byte
branch
2 times, most recently
from
July 11, 2026 20:02
4d55387 to
f3979b0
Compare
ESRogs
changed the base branch from
json-string-escape-decoding
to
json-scalar-tail
July 11, 2026 20:02
ESRogs
force-pushed
the
split-at-utf8-byte
branch
from
July 11, 2026 20:39
f3979b0 to
937c99f
Compare
ESRogs
force-pushed
the
json-scalar-tail
branch
2 times, most recently
from
July 12, 2026 17:24
19448c0 to
1bef00c
Compare
ESRogs
force-pushed
the
split-at-utf8-byte
branch
from
July 12, 2026 17:24
937c99f to
4d1b851
Compare
ESRogs
force-pushed
the
split-at-utf8-byte
branch
from
July 12, 2026 17:26
4d1b851 to
b1536b2
Compare
ESRogs
force-pushed
the
split-at-utf8-byte
branch
from
July 12, 2026 18:55
b1536b2 to
ac98fec
Compare
ESRogs
force-pushed
the
json-scalar-tail
branch
2 times, most recently
from
July 13, 2026 08:07
f0949e5 to
00ecbe1
Compare
ESRogs
force-pushed
the
split-at-utf8-byte
branch
2 times, most recently
from
July 13, 2026 08:45
8fd9f05 to
8798428
Compare
ESRogs
force-pushed
the
json-scalar-tail
branch
2 times, most recently
from
July 13, 2026 09:00
50d1a04 to
6b4e69a
Compare
ESRogs
force-pushed
the
split-at-utf8-byte
branch
2 times, most recently
from
July 13, 2026 09:08
3179d60 to
7798499
Compare
ESRogs
force-pushed
the
split-at-utf8-byte
branch
from
July 13, 2026 20:50
7798499 to
66dc1a7
Compare
ESRogs
force-pushed
the
split-at-utf8-byte
branch
from
July 13, 2026 23:56
66dc1a7 to
f83a27a
Compare
ESRogs
force-pushed
the
split-at-utf8-byte
branch
from
July 13, 2026 23:57
f83a27a to
e1520e3
Compare
ESRogs
force-pushed
the
split-at-utf8-byte
branch
from
July 13, 2026 23:58
e1520e3 to
35dd19a
Compare
ESRogs
force-pushed
the
split-at-utf8-byte
branch
from
July 14, 2026 00:00
35dd19a to
9acd732
Compare
ESRogs
force-pushed
the
split-at-utf8-byte
branch
from
July 14, 2026 00:08
9acd732 to
2e26eda
Compare
ESRogs
force-pushed
the
split-at-utf8-byte
branch
from
July 14, 2026 00:12
2e26eda to
448911b
Compare
…ntrol bytes Pin the rejected control range at both ends: raw 0x00 and raw 0x1F in a string reject the document, not just an interior control byte. Add a positive test for raw DEL (0x7F) — just past the control range and valid unescaped — and verify that a raw control byte inside a *skipped* string also rejects the document. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
split_json_scalar_tail made JSON parsing quadratic — more precisely, O(num scalars * document size): it ran Str.find_first once per possible delimiter (seven scans, each running to the first occurrence of its delimiter or to the end of the document), and it rebuilt `after` by copying the remaining document into a fresh allocation, up to seven times per scalar. Find the first delimiter in a single pass instead (List.find_first_index with an is_json_scalar_delimiter predicate) and return both halves as zero-copy slices, keeping the delimiter in `after`. Parsing becomes linear and scalar splitting allocates nothing: a loop of 20k parses of a 500-field numeric document, which previously aborted the no-alloc json-decoder test platform, completes in ~1.5s (~75µs per parse, LLVM --opt=speed) — the same band as the equivalent string-valued document (~1.4s). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The platform's host aborts on any runtime roc_alloc, but its runtime documents were all string-valued, and app.roc's hardcoded numeric parses are evaluated at compile time — so scalar skipping was never exercised under the allocation trap, and split_json_scalar_tail's previously allocating implementation went uncaught. Add a skipped numeric field to the runtime-built document so the trap covers scalars permanently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Negative-path coverage for scalar parsing against the RFC 8259 grammar: exact literal spellings (wrong case, truncation, trailing junk), number reject paths (dangling decimal points, incomplete exponents, lone minus, leading plus, leading zeros, Infinity, hex, junk after a valid number), accept paths that had no tests (uppercase and signed exponent markers, sign+fraction+exponent combined, negative zero per target type), and scalar tokenization at every delimiter (each JSON whitespace byte, comma, closing brace, closing bracket, and the empty-scalar reject). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Splits a Str at a byte offset into zero-copy before/after slices:
split_at_utf8_byte : Str, U64
-> Try({ before : Str, after : Str }, [OutOfBounds, NotACharacterBoundary])
The offset counts UTF-8 bytes (as in Str.count_utf8_bytes) and must be
at most the string's length and fall on a character boundary — the
start of a code point, or the end of the string — so a valid split
always yields two valid Strs.
The compiler-implemented str_split_at_utf8_byte_raw performs both
checks in the shared Zig runtime and returns a flags record
({ before, after, is_not_char_boundary, is_out_of_bounds } — each
flag names one failure, and both false means success) that the public
wrapper maps into Try, so no unchecked operation is expressible from
Roc. Wired through: LowLevel enum (+ retainsSharingArgs group),
BuiltinLowLevel canonicalization, str.zig runtime, dev_wrappers C-ABI
wrapper and layout struct, both static-lib export tables, the dev
object reader, the dev, LLVM, and wasm backends, the wasm runner host
shim, and the interpreter.
The JSON string scanner now splits at the closing quote with the
intrinsic instead of reconstructing slices via from_utf8 +
drop_prefix, making the split O(1) in the string body instead of
O(offset). Interleaved LLVM benchmarks against the parent commit,
each figure the median wall time for a loop of 20k Json.parse calls
(default --opt=speed, macOS arm64): 500 short string fields
1.43s -> 1.37s, 3 x 4.9KB fields 0.17s -> 0.15s, escape-bearing
fields 1.54s -> 1.48s, numeric fields (only the short keys hit the
scanner) 1.50s -> 1.49s. On the dev backend, which does not inline,
the intrinsic call is a small per-string overhead instead; the win
applies to optimized builds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ESRogs
force-pushed
the
split-at-utf8-byte
branch
from
July 14, 2026 00:21
448911b to
6ae2447
Compare
ESRogs
force-pushed
the
json-scalar-tail
branch
4 times, most recently
from
July 16, 2026 01:11
54e2b92 to
61fa4f5
Compare
This was referenced Jul 18, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a public
Str.split_at_utf8_byte:Splits a
Strat a byte offset into zero-copybefore/afterslices. The offset counts UTF-8 bytes (as inStr.count_utf8_bytes) and must be at most the string's length and on a character boundary, so a valid split always yields two validStrs. The backing intrinsic (str_split_at_utf8_byte_raw, following thestr_find_first_rawconvention) performs both checks in the shared Zig runtime and returns a flags record —is_out_of_boundsandis_not_char_boundary, each naming one failure, both false on success — that the wrapper maps intoTry. No unchecked operation is expressible from Roc.The JSON string scanner (from roc-lang#10045) uses it to split at the closing quote instead of reconstructing slices via
from_utf8+drop_prefix: O(1) in the string body instead of O(offset).Performance (interleaved LLVM benchmarks vs the parent commit; each cell is the median wall time for a loop of 20,000
Json.parsecalls, default--opt=speed, macOS arm64):Two measured boundaries on where the win applies:
Open questions for iteration:
split_at_utf8_bytefor thecount_utf8_bytesrhyme.Trywith[OutOfBounds, NotACharacterBoundary]publicly; two named failure flags at the intrinsic layer (a comment on the Zig result struct notes that accumulating more failure modes should trigger a switch tofrom_utf8-styleis_ok+ problem code).Strat all — to be raised on Zulip #API design with this diff as the concrete proposal.