Skip to content

v0.26.0

Choose a tag to compare

@rustyconover rustyconover released this 21 Jul 17:38
· 276 commits to main since this release

Client-authoritative HTTP codec negotiation

Response codecs are now chosen by walking the client's stated preference — X-VGI-Accept-Encoding ahead of the generic Accept-Encoding — rather than a server-side order.

cpp-httplib, the DuckDB extension's HTTP client, injects Accept-Encoding: deflate, gzip, br, zstd. Honouring that gzip-first order instead of VGI's own zstd-first preference cost 432 ms vs ~40 ms of zstd on 200 MB of Arrow bodies — a 4.2x slower round-trip.

A codec reachable only via the custom header is stamped on X-VGI-Content-Encoding, since such a client's fetch layer would mangle or auto-decode a standard Content-Encoding. This is the browser path: fetch() cannot set Accept-Encoding at all.

  • identity is a first-class token. A client can explicitly demand an uncompressed body, and Content-Encoding: identity on a request now passes through rather than returning 415.
  • VGI-Supported-Encodings is always emitted, including present-but-empty. Absent means "legacy server, assume zstd"; empty means "speaks no compression". Omitting it made a deliberately-uncompressed server indistinguishable from an old one, and got it sent zstd bodies it could not decode. The client learns the same distinction.

Changed default

compression_level drops from 3 to 1. On Arrow IPC bodies level 1 measured 4.4x faster and smaller on hex-digest data, 2.7x faster and smaller on mixed numeric, and 1.4x faster for 2% more bytes on compressible text. All five VGI SDKs standardize on it.

Performance

  • Nested dataclass deserialization consumes the cached per-class plan instead of re-resolving type hints per call: 45.9 → 32.5 µs (−29%). This was the one hot site 0.25.0's plan cache missed, and it sits on the HTTP state-token path.
  • waitress buffers sized for Arrow. Its 512 KiB inbuf_overflow spooled larger request bodies to a temp file on disk and read them back. 8 MiB round-trip: 25.5 → 9.1 ms (2.81x).
  • VGI_RPC_IPC_VALIDATION (none/standard/full) makes validation a deploy-time knob. Default stays FULL; an explicit argument still wins, and an invalid value warns and falls back to FULL — a typo can only ever yield more validation than intended. Measured on a 6.38 MB batch: full 0.571 ms, standard and none both ~0.
  • serve_http gained max_request_bytes and compression_level; it previously could not advertise a request cap or disable compression at all.

Conformance

TestHttpCompressionNegotiationConformance covers the negotiation rules cross-language, capability-gated on the advertised codec set. The conformance worker gained --no-compression and a conformance_http_no_compression_port fixture, so the empty-advertisement case runs against a real compression-disabled server. The suite resolves that fixture indirectly, so SDK runners that lack it skip rather than error at collection.