v0.31.0 — call/cursor token split, compact state codec, per-platform AEAD
Performance work on the HTTP streaming path. Measured end to end on a
loaded server (waitress, 8 concurrent clients): 987 → 1429 turns/s, with
per-turn GIL-held time falling 730µs → 451µs.
Stream state is now two tokens, not one
A stream's state divides into a half fixed for the life of the call and a
half that advances per turn. Carrying both in one token meant every
continuation re-serialized, re-sealed, re-opened and re-parsed the fixed
part — 7,224 of 8,344 state bytes on a plain scan.
CALL_STATE_KEY— minted once by/init, echoed by the client, never
re-issued. Servers keep a bounded per-process cache, so a warm process
skips opening it entirely (measured 100% hit rate, including across four
worker processes).STATE_KEY— the cursor, re-minted per turn, and the only token a
response returns.
The cursor token is opened first and its AEAD tag covers the call id, so
only an authenticated id ever reaches a cache lookup. A miss falls back to
the client's call token, keeping the transport stateless.
Flat state skips Arrow IPC
A one-row Arrow stream pays for a schema message, a batch message, an
end-of-stream marker and padding. For a two-int cursor that framing is the
cost: 416 bytes and 36µs, against 16 bytes and 0.21µs packed directly. Flat
dataclasses now use a positional binary encoding; anything Arrow is
genuinely for keeps the Arrow path, dispatched on the payload's first byte.
Tokens are compressed inside the seal
It has to be that way round — once sealed a token is ciphertext, so the body
codec finds nothing. Compressing inside reaches the real redundancy: a
7,800-byte call state packs to 1,872. Per-turn wire traffic 15,621 → 5,567
bytes, at no measurable CPU cost.
AEAD backend is chosen per platform
Neither library wins everywhere. PyCryptodome builds a cipher object in
Python per call; PyNaCl is one C call into a libsodium whose speed depends
on the wheel's SIMD dispatch. At our token size: macOS arm64 14.5µs vs
17.0µs, Linux x86_64 37.1µs vs 6.3µs. PyNaCl is now declared only where it
measured faster, so the resolver makes the choice — no runtime probing.
Both emit byte-identical XChaCha20-Poly1305 envelopes, asserted by tests, so
this is safe to vary across a fleet.
waitress defaults corrected
- Thread default 16 → 4. 16 is dominated: worse than 4 for blocking
functions and worse than 1 for computing ones. - Waitress's per-request queue-depth warning is quieted (it was 5.7% of
GIL-held time);VGI_RPC_WAITRESS_QUEUE_LOG=1restores it. - Its Arrow buffer tuning now has one definition instead of three drifting
copies.
Also
empty_batchcached per schema — worth 43% of a turn at 2000 columns.- IPC read options shared; bytes handed to pyarrow directly rather than
wrapped in aBufferReader. - One-row batches built column-by-column, reusing a cached null array per
column (3.8× on a 17-column dataclass). - Per-request dispatch imports resolved once.