Skip to content

v0.31.0 — call/cursor token split, compact state codec, per-platform AEAD

Choose a tag to compare

@rustyconover rustyconover released this 03 Aug 19:37
· 233 commits to main since this release

Performance work on the HTTP streaming path. Measured end to end on a
loaded server (waitress, 8 concurrent clients): 987 → 1429 turns/s, with
per-turn GIL-held time falling 730µs → 451µs.

Stream state is now two tokens, not one

A stream's state divides into a half fixed for the life of the call and a
half that advances per turn. Carrying both in one token meant every
continuation re-serialized, re-sealed, re-opened and re-parsed the fixed
part — 7,224 of 8,344 state bytes on a plain scan.

  • CALL_STATE_KEY — minted once by /init, echoed by the client, never
    re-issued. Servers keep a bounded per-process cache, so a warm process
    skips opening it entirely (measured 100% hit rate, including across four
    worker processes).
  • STATE_KEY — the cursor, re-minted per turn, and the only token a
    response returns.

The cursor token is opened first and its AEAD tag covers the call id, so
only an authenticated id ever reaches a cache lookup. A miss falls back to
the client's call token, keeping the transport stateless.

Flat state skips Arrow IPC

A one-row Arrow stream pays for a schema message, a batch message, an
end-of-stream marker and padding. For a two-int cursor that framing is the
cost: 416 bytes and 36µs, against 16 bytes and 0.21µs packed directly. Flat
dataclasses now use a positional binary encoding; anything Arrow is
genuinely for keeps the Arrow path, dispatched on the payload's first byte.

Tokens are compressed inside the seal

It has to be that way round — once sealed a token is ciphertext, so the body
codec finds nothing. Compressing inside reaches the real redundancy: a
7,800-byte call state packs to 1,872. Per-turn wire traffic 15,621 → 5,567
bytes, at no measurable CPU cost.

AEAD backend is chosen per platform

Neither library wins everywhere. PyCryptodome builds a cipher object in
Python per call; PyNaCl is one C call into a libsodium whose speed depends
on the wheel's SIMD dispatch. At our token size: macOS arm64 14.5µs vs
17.0µs, Linux x86_64 37.1µs vs 6.3µs. PyNaCl is now declared only where it
measured faster, so the resolver makes the choice — no runtime probing.
Both emit byte-identical XChaCha20-Poly1305 envelopes, asserted by tests, so
this is safe to vary across a fleet.

waitress defaults corrected

  • Thread default 16 → 4. 16 is dominated: worse than 4 for blocking
    functions and worse than 1 for computing ones.
  • Waitress's per-request queue-depth warning is quieted (it was 5.7% of
    GIL-held time); VGI_RPC_WAITRESS_QUEUE_LOG=1 restores it.
  • Its Arrow buffer tuning now has one definition instead of three drifting
    copies.

Also

  • empty_batch cached per schema — worth 43% of a turn at 2000 columns.
  • IPC read options shared; bytes handed to pyarrow directly rather than
    wrapped in a BufferReader.
  • One-row batches built column-by-column, reusing a cached null array per
    column (3.8× on a 17-column dataclass).
  • Per-request dispatch imports resolved once.