fix: Improve jreader throughput with a single-pass tokenizer

[kinyoklion]

SDK-2884

The parse-side counterpart of #51. jreader's default tokenizer scanned input character by character: readString walked every character through bytes.Reader.ReadRune (a method call plus UTF-8 decode per character, even for plain ASCII — the single hottest function when parsing a real flag payload), escaped strings were built by appending one rune at a time, and scalar reads shuttled 48-byte token structs through two call layers.

This PR rewrites the tokenizer as a single-pass scanner over the input byte slice:

• Strings scan in place. Unescaped strings keep the existing zero-copy behavior (the returned bytes are a subslice of the input, sliced identically to before); since only " and \ end the scan, plain strings — including non-ASCII ones — are scanned without any UTF-8 decoding. Strings containing escapes decode in one forward pass into a single buffer sized up front, bulk-copying clean runs, with \uXXXX decoded by direct indexing. bytes.Reader and per-rune appends are gone.
• Scalar reads (Bool, Number, String, PropertyName, Any) dispatch on the first non-whitespace byte and parse directly, entering the token path only for pushed-back tokens or type mismatches, so all error construction still goes through the same primitives. next() parses into a reused field on the tokenReader instead of returning token structs; putBack is a flag flip.
• A bytes.IndexByte-based variant (find the closing quote, then scan the span) was benchmarked and not taken: ~6x faster on 400-byte strings but 15-20% slower on short ones, and on the real flag payload the two were statistically indistinguishable (p=1.000), so the simpler loop stays.

This is strictly a performance change, rebased onto the #55 revert: with the baseline deliberately kept lenient to minimize behavioral change, the rewrite preserves the current implementation's observable behavior exactly — exported API, parsed values, error types, error message strings, and error offsets are all identical. That includes the baseline's deliberate looseness, verified case by case:

• Whitespace between tokens is unicode.IsSpace applied per byte, so raw 0x0B, 0x0C, 0x85, and 0xA0 bytes are accepted as whitespace.
• Unescaped strings pass control characters and invalid UTF-8 through verbatim; strings containing an escape re-encode each invalid byte as U+FFFD (the existing asymmetry).
• \uXXXX surrogate code points encode as U+FFFD without pair combining, exactly as the current appendRune does.
• Number scanning is the same lenient scan, including its exact input positions after a malformed number (observable through RequireEOF), and integer literals beyond int64 wrap exactly as before.

Correctness validation

Beyond the existing suite (green with -race; the cross-permutation commontest matrices cover both dispatch paths), a differential harness ran the current (37b14d6) and new tokenizers in lock-step over 1,020,486 paired runs across 145,464 inputs with zero mismatches: exhaustive one- and two-byte string contents, all 65,536 \uXXXX values, surrogate grids, invalid-UTF-8 sequences at eight positions, all 256 single bytes in each inter-token position (for the per-byte whitespace rule), int64 extremes, -0.0, denormals, overflow exponents, 400-digit literals, malformed numbers, every prefix-truncation of a mixed document, and random documents with trailing junk and byte mutations. Comparisons covered decoded values (bit-exact floats, including wrapped int64 literals), string bytes and nilness, error type/fields/message/offset, RequireEOF, and failed-state stickiness. The harness's sensitivity was proven by seeded mutations: an ASCII-only whitespace table produced 204 mismatches, and surrogate pair combining produced 11,806. The edge cases the in-repo suites did not reach are now pinned by table-driven tests (exact error type, message, value, and offset), bringing token_reader_default.go to 100% statement coverage.

Benchmarks

linux/amd64, interleaved old/new binaries (3 rounds x count 2, benchstat n=6, all listed deltas significant at p<=0.002). Micro geomean -56.7% vs the current v4 baseline:

ReadBoolean                    -42.9%
ReadNumberInt                  -47.9%
ReadString                     -55.7%
ReadArrayOfStrings             -52.2%   (allocs 208 -> 158)
ReadStringKinds/shortASCII     -71%
ReadStringKinds/longASCII      -84%
ReadStringKinds/multiByte      -75.9%
ReadStringKinds/escaped        -63.8%   (allocs 101 -> 51)
ReadObjectNoAlloc              -54.0%
ReadArrayOfObjects             -51.0%

On a real 3,228-flag / 3.2 MB LaunchDarkly payload parsed through ldmodel (go-server-sdk-evaluation): direct jreader parse 38.6 ms -> 19.1 ms (-50.5%); through the encoding/json-dispatch path, 70.0 ms -> 50.2 ms (-28.2%). For reference, the v3 easyjson lexer's advantage over the old default reader on the same payload was ~19% — relevant because v4 removed the easyjson adapters, and this closes that gap with margin.

All NoAlloc benchmarks remain at 0 allocs/op (CI gate).

Notes for reviewers

• The first-byte dispatch in any() and the pushed-back-token path (tokenToAnyValue) are parallel switches that must stay in sync; both are covered by the commontest permutation matrices and the differential corpus.
• readNumber's consume/unread positions on malformed input (e.g. where the scan stops in 1ex vs 1e+x) are the subtlest part of the port; they are reproduced exactly and covered by the differential corpus.
• decodedStringCapacity pre-scans an escaped string's remainder once to size the decode buffer — a deliberate two-scans/one-allocation trade.
• Number parsing still allocates one string per float (strconv.ParseFloat); avoiding it would require unsafe, which this repo does not use.

---

Note

Overview
Rewrites the default jreader tokenizer as a single-pass byte scanner, roughly doubling parse throughput on real LaunchDarkly payloads while preserving observable behavior.

Strings no longer walk every character through bytes.Reader.ReadRune. Unescaped strings stay zero-copy subslices; escaped strings decode in one forward pass into a pre-sized buffer with bulk copies of plain runs. Scalar reads (Bool, Number, String, Any) dispatch on the first non-whitespace byte and parse directly, falling back to the token path only for pushed-back tokens or type mismatches. next() writes into a reused tok field instead of returning 48-byte token structs.

Adds lookup tables for whitespace and plain string bytes, new edge-case tests covering malformed numbers/strings and pushed-back tokens, and BenchmarkReadStringKinds for the distinct string paths.

^{Reviewed by Cursor Bugbot for commit 63ffa7c. Bugbot is set up for automated code reviews on this repo. Configure here.}