Skip to content

v0.3.1

Choose a tag to compare

@ehsanmok ehsanmok released this 11 Sep 04:17
· 51 commits to main since this release

A patch release, but a substantial one on the write path. Parsing is unchanged; everything here is about building and serializing JSON, plus correctness fixes. The public API is unchanged for users on v0.3.0 — existing code gets the speedups without edits.

Fixes

  • Control characters are now escaped everywhere. A string containing U+0000–U+001F inside a hand-built container serialized to raw bytes, which RFC 8259 §7 forbids, while the same value read from a parser serialized correctly. Escaping existed in four copies that had drifted apart; they now share one implementation. This had no test coverage before; it does now.
  • pixi install works again (#3). Pinning pixi-build-rattler-build to an exact version hard-coupled the repo to one pixi release: that backend speaks build-api-version 4, while every pixi after 0.70.2 provides ≥ 5, so resolution failed before anything could build — on both osx-arm64 and linux-64. The pin is now a range, and CI no longer pins pixi either.
  • The conda package builds on macOS again. simdjson 4.6.11 uses std::inserter without including <iterator>; libstdc++ pulls it in transitively but libc++ does not, so pixi build failed on macOS for anyone consuming this library as a source package. The FFI wrapper now includes <iterator> itself, so it is independent of whichever simdjson the solver picks. CI now runs pixi build on both platforms — the test suite never exercised recipe.yaml's build script, which is why this was invisible here and only surfaced downstream.

Performance

Building a Value tree was O(n²): set / append / set_at materialized the whole tree and rebuilt the document tape on every call. Value now holds either a tape-backed view or an owned tree and mutates the latter in place, converting at most once. Serialization gained a buffered writer, so a leaf's bytes are copied once rather than once per nesting level.

Measured on a nested 47 KB document (100 records), best of 15:

Operation v0.3.0 v0.3.1
Build a tree — same code as v0.3.0 20.1 ms 1.3 ms
Build a tree — via Value.object() / Value.array() — 0.79 ms
Build a tree — transferring containers with ^ — 0.41 ms
dumps 425 µs 135 µs
serialize_json (typed, no intermediate tree) — 40 µs
deserialize_json (small nested struct) 4.9 µs 1.1 µs
loads 91 µs 91 µs

Typed deserialization was reparsing: every field read serialized the child subtree back to a string and ran a full parse over it to recover one scalar, repeating at each level of nesting. It now indexes the parsed value directly.

Parse throughput is unchanged — 1.06 GB/s on twitter.json, 1.56 GB/s on citm_catalog.json (x86, parse_only).

New

  • JsonWriter is public. It is the buffered byte sink every serializer emits into: one pre-sized buffer, a SIMD scan that skips escaping when it is not needed, and a table-driven integer path. When the shape is known ahead of time, writing into it directly is the fastest route out.
  • Reflection serializes any list element type. List[<struct>] previously emitted List's internal fields as an object; one generic path now covers lists of scalars, of structs, and nested lists. All sized integer widths (Int8…Int64, UInt8…UInt64) are supported in both directions. Deserializing a list of structs remains unsupported and now says so explicitly.
  • Value.object() / Value.array() build a container without going through the parser. Scalar Values are allocation-free.

Notes

set / append / set_at deep-copy the subtree they are given unless you hand over ownership. For large trees, transfer with ^:

var items = Value.array()
for row in rows:
    items.append(build_row(row))   # temporary: moved, no copy
doc.set("items", items^)           # named local: `^` avoids a deep copy

Reproduce the write-path numbers with pixi run -e dev bench-build.

Full Changelog: v0.3.0...v0.3.1