Walk JSON strings in place instead of materializing byte lists#5
Closed
ESRogs wants to merge 1 commit into
Closed
Conversation
ESRogs
commented
Jul 16, 2026
ESRogs
commented
Jul 16, 2026
ESRogs
force-pushed
the
json-string-str-walk
branch
9 times, most recently
from
July 16, 2026 17:17
83ad44a to
76176b7
Compare
ESRogs
force-pushed
the
str-utf8-iteration-json-scanning
branch
from
July 18, 2026 17:15
0b9fe5e to
16dc53d
Compare
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ESRogs
force-pushed
the
json-string-str-walk
branch
from
July 18, 2026 17:18
76176b7 to
e370d74
Compare
Owner
Author
|
Combined with #8 and opened upstream as roc-lang#10217 (branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
This PR is stacked on roc-lang#10173, which should merge first. It applies roc-lang#10173's UTF-8 primitives to the one JSON scanning path that PR leaves on the old representation. (It supersedes the approach in #1, which proposed a dedicated split intrinsic for the same call site — kept open for reference.)
Overview
roc-lang#10173 rewrites JSON whitespace trimming and scalar splitting to walk the
Strdirectly viastr_get_utf8_byte_unsafe, eliminating theStr.to_utf8List materialization. This PR completes the pattern for the remaining JSON string path: the string scanner (scan_json_string_tail), the string-body decoder (decode_json_string_body), and the escape parser they share. After it, no JSON string path convertsStrtoList(U8). (Two scalar validators —is_json_numberandjson_dec_digits_are_zero— are the parse path's remainingto_utf8users; converting them is a natural follow-up.)Detail
scan_json_string_tailwalks the string's bytes in place. At the closing quote, the body and the text after the quote are produced with twostr_substring_unsafecalls — the walk stopped on an ASCII quote, so both cut points are proven UTF-8 boundaries. This replaces the previous split, which paid two O(index) passes per string:Str.from_utf8(List.take_first(bytes, index))revalidated the whole prefix, andStr.drop_prefix(tail, raw)byte-compared that same prefix again to recover the tail.decode_json_string_bodywalks the same way. Its output allocation (building the decoded string) is inherent and unchanged; its input walk no longer materializes a List.parse_json_escape_sequence— the escape grammar shared by both callers — keeps itsrest-shaped signature, now asStr -> …: callers pass the bytes after the backslash as a zero-copystr_substring_unsafeslice, and it reads them viastr_get_utf8_byte_unsafewith explicit bounds checks in place of list patterns. One grammar, two callers, as before. (Benchmarked against an offset-parameter variant: no measurable difference, so the self-contained signature wins.)str_drop_first_bytes_unsafe(private) is the shared slice helper: dropcountbytes from the front as a zero-copy slice, clamping to""past the end, with the UTF-8 boundary as a caller guarantee. The scanner and decoder call it directly at their proven-ASCII cut points, andStr.drop_first_bytesis refactored to validate-then-delegate to it, so the checked and unchecked versions share one slicing implementation (LLVM folds the wrapper's duplicated length compare).Allocation coverage
test/alloc-count/app.rocgainscheck_json_parse_allocs!: parsing a small (SSO) document with a clean string field performs zero allocations, with the string field decoded or skipped. Before this PR the scan of an SSO document paidStr.to_utf8's copy (the platform's control check pins that copy at exactly 1 allocation). Heap-backed documents already shared their buffer throughto_utf8, so the allocation win is specifically for small inputs; the throughput win below is general.Benchmarks
Interleaved against the base branch (roc-lang#10173's tip; median wall time for a loop of 20,000
Json.parsecalls decoding{ k0 : Str }, default LLVM--opt=speed, macOS arm64):Testing
roc testsuites:JsonStringEscapes(64),JsonScalarParseEdgeCases(51),JsonEncodeRoundTrip(27) — all pass; the escape suite pins the ported grammar (named escapes,\uXXXX, surrogate pairs, truncations) at every grammar position.--opt=speedwith the new zero-allocation checks.