v0.3.1
A patch release, but a substantial one on the write path. Parsing is unchanged; everything here is about building and serializing JSON, plus correctness fixes. The public API is unchanged for users on v0.3.0 — existing code gets the speedups without edits.
Fixes
- Control characters are now escaped everywhere. A string containing
U+0000–U+001Finside a hand-built container serialized to raw bytes, which RFC 8259 §7 forbids, while the same value read from a parser serialized correctly. Escaping existed in four copies that had drifted apart; they now share one implementation. This had no test coverage before; it does now. pixi installworks again (#3). Pinningpixi-build-rattler-buildto an exact version hard-coupled the repo to one pixi release: that backend speaks build-api-version 4, while every pixi after 0.70.2 provides ≥ 5, so resolution failed before anything could build — on bothosx-arm64andlinux-64. The pin is now a range, and CI no longer pins pixi either.- The conda package builds on macOS again. simdjson 4.6.11 uses
std::inserterwithout including<iterator>; libstdc++ pulls it in transitively but libc++ does not, sopixi buildfailed on macOS for anyone consuming this library as a source package. The FFI wrapper now includes<iterator>itself, so it is independent of whichever simdjson the solver picks. CI now runspixi buildon both platforms — the test suite never exercisedrecipe.yaml's build script, which is why this was invisible here and only surfaced downstream.
Performance
Building a Value tree was O(n²): set / append / set_at materialized the whole tree and rebuilt the document tape on every call. Value now holds either a tape-backed view or an owned tree and mutates the latter in place, converting at most once. Serialization gained a buffered writer, so a leaf's bytes are copied once rather than once per nesting level.
Measured on a nested 47 KB document (100 records), best of 15:
| Operation | v0.3.0 | v0.3.1 |
|---|---|---|
| Build a tree — same code as v0.3.0 | 20.1 ms | 1.3 ms |
Build a tree — via Value.object() / Value.array() |
— | 0.79 ms |
Build a tree — transferring containers with ^ |
— | 0.41 ms |
dumps |
425 µs | 135 µs |
serialize_json (typed, no intermediate tree) |
— | 40 µs |
deserialize_json (small nested struct) |
4.9 µs | 1.1 µs |
loads |
91 µs | 91 µs |
Typed deserialization was reparsing: every field read serialized the child subtree back to a string and ran a full parse over it to recover one scalar, repeating at each level of nesting. It now indexes the parsed value directly.
Parse throughput is unchanged — 1.06 GB/s on twitter.json, 1.56 GB/s on citm_catalog.json (x86, parse_only).
New
JsonWriteris public. It is the buffered byte sink every serializer emits into: one pre-sized buffer, a SIMD scan that skips escaping when it is not needed, and a table-driven integer path. When the shape is known ahead of time, writing into it directly is the fastest route out.- Reflection serializes any list element type.
List[<struct>]previously emittedList's internal fields as an object; one generic path now covers lists of scalars, of structs, and nested lists. All sized integer widths (Int8…Int64,UInt8…UInt64) are supported in both directions. Deserializing a list of structs remains unsupported and now says so explicitly. Value.object()/Value.array()build a container without going through the parser. ScalarValues are allocation-free.
Notes
set / append / set_at deep-copy the subtree they are given unless you hand over ownership. For large trees, transfer with ^:
var items = Value.array()
for row in rows:
items.append(build_row(row)) # temporary: moved, no copy
doc.set("items", items^) # named local: `^` avoids a deep copyReproduce the write-path numbers with pixi run -e dev bench-build.
Full Changelog: v0.3.0...v0.3.1