Skip to content

Walk JSON strings in place instead of materializing byte lists#5

Closed
ESRogs wants to merge 1 commit into
str-utf8-iteration-json-scanningfrom
json-string-str-walk
Closed

Walk JSON strings in place instead of materializing byte lists#5
ESRogs wants to merge 1 commit into
str-utf8-iteration-json-scanningfrom
json-string-str-walk

Conversation

@ESRogs

@ESRogs ESRogs commented Jul 16, 2026

Copy link
Copy Markdown
Owner

Important

This PR is stacked on roc-lang#10173, which should merge first. It applies roc-lang#10173's UTF-8 primitives to the one JSON scanning path that PR leaves on the old representation. (It supersedes the approach in #1, which proposed a dedicated split intrinsic for the same call site — kept open for reference.)

Overview

roc-lang#10173 rewrites JSON whitespace trimming and scalar splitting to walk the Str directly via str_get_utf8_byte_unsafe, eliminating the Str.to_utf8 List materialization. This PR completes the pattern for the remaining JSON string path: the string scanner (scan_json_string_tail), the string-body decoder (decode_json_string_body), and the escape parser they share. After it, no JSON string path converts Str to List(U8). (Two scalar validators — is_json_number and json_dec_digits_are_zero — are the parse path's remaining to_utf8 users; converting them is a natural follow-up.)

Detail

  • scan_json_string_tail walks the string's bytes in place. At the closing quote, the body and the text after the quote are produced with two str_substring_unsafe calls — the walk stopped on an ASCII quote, so both cut points are proven UTF-8 boundaries. This replaces the previous split, which paid two O(index) passes per string: Str.from_utf8(List.take_first(bytes, index)) revalidated the whole prefix, and Str.drop_prefix(tail, raw) byte-compared that same prefix again to recover the tail.

  • decode_json_string_body walks the same way. Its output allocation (building the decoded string) is inherent and unchanged; its input walk no longer materializes a List.

  • parse_json_escape_sequence — the escape grammar shared by both callers — keeps its rest-shaped signature, now as Str -> …: callers pass the bytes after the backslash as a zero-copy str_substring_unsafe slice, and it reads them via str_get_utf8_byte_unsafe with explicit bounds checks in place of list patterns. One grammar, two callers, as before. (Benchmarked against an offset-parameter variant: no measurable difference, so the self-contained signature wins.)

  • str_drop_first_bytes_unsafe (private) is the shared slice helper: drop count bytes from the front as a zero-copy slice, clamping to "" past the end, with the UTF-8 boundary as a caller guarantee. The scanner and decoder call it directly at their proven-ASCII cut points, and Str.drop_first_bytes is refactored to validate-then-delegate to it, so the checked and unchecked versions share one slicing implementation (LLVM folds the wrapper's duplicated length compare).

Allocation coverage

test/alloc-count/app.roc gains check_json_parse_allocs!: parsing a small (SSO) document with a clean string field performs zero allocations, with the string field decoded or skipped. Before this PR the scan of an SSO document paid Str.to_utf8's copy (the platform's control check pins that copy at exactly 1 allocation). Heap-backed documents already shared their buffer through to_utf8, so the allocation win is specifically for small inputs; the throughput win below is general.

Benchmarks

Interleaved against the base branch (roc-lang#10173's tip; median wall time for a loop of 20,000 Json.parse calls decoding { k0 : Str }, default LLVM --opt=speed, macOS arm64):

document base this PR
500 short string fields 1.21 s 0.97 s (-20%)
3 × 4.9 KB string fields 0.19 s 0.17 s (-10%)
300 escape-bearing fields (validated during skip) 0.95 s 0.75 s (-21%)
500 numeric fields (only the short keys hit the scanner) 1.22 s 1.14 s (-7%)

Testing

  • roc test suites: JsonStringEscapes (64), JsonScalarParseEdgeCases (51), JsonEncodeRoundTrip (27) — all pass; the escape suite pins the ported grammar (named escapes, \uXXXX, surrogate pairs, truncations) at every grammar position.
  • The alloc-count platform app passes on the dev backend and --opt=speed with the new zero-allocation checks.

Comment thread src/build/roc/Builtin.roc Outdated
Comment thread src/build/roc/Builtin.roc Outdated
@ESRogs
ESRogs force-pushed the json-string-str-walk branch 9 times, most recently from 83ad44a to 76176b7 Compare July 16, 2026 17:17
@ESRogs
ESRogs force-pushed the str-utf8-iteration-json-scanning branch from 0b9fe5e to 16dc53d Compare July 18, 2026 17:15
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ESRogs
ESRogs force-pushed the json-string-str-walk branch from 76176b7 to e370d74 Compare July 18, 2026 17:18
@ESRogs

ESRogs commented Jul 18, 2026

Copy link
Copy Markdown
Owner Author

Combined with #8 and opened upstream as roc-lang#10217 (branch json-parse-no-byte-lists). Kept for the review history.

@ESRogs ESRogs closed this Jul 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant