Skip to content

v0.33.0 — protocol wire name vgi.v2

Choose a tag to compare

@rustyconover rustyconover released this 17 Sep 16:34
· 206 commits to main since this release

Breaking: the wire protocol name is now vgi.v2

The transports require a vgi_rpc.protocol routing key, and this port never
called SetServiceName — so it hosted the framework's default literal
Service and answered ProtocolNotSupportedError to any client sending a real
name. Six implementations had four different answers. The canonical name is now
vgi.v2, declared here as the exported constant vgi.ProtocolName and passed
to SetServiceName on every transport the worker builds.

POST /Service/bind   -> does not host protocol "Service". Hosted: [vgi.v2]
POST /bogus.v1/bind  -> does not host protocol "bogus.v1". Hosted: [vgi.v2]
POST /vgi.v2/bind    -> routes

The major is in the name deliberately. An incompatible major becomes a
different name, so a mismatched client gets a routing 404 that any proxy,
gateway or log already understands, rather than a mysterious deserialization
failure deep in a stream. It also means vgi.v2 can be served beside a future
vgi.v3 while clients migrate.

What you must do to upgrade. Nothing in your worker code — the name is set
by the SDK. But the client has to send it: the DuckDB extension must be new
enough to send the vgi_rpc.protocol routing key (Query-farm/vgi 5d45bb4 and
later). A build carrying that is published to the community channel, so
FORCE INSTALL vgi FROM community; is the upgrade path. An older extension
paired with this release — or this release's predecessor paired with a current
extension — fails at bind, not at first data.

This port does not host the secret protocol (vgi.secret.v1); secrets travel as
fields inside the vgi protocol's own requests.

Transport: vgi-rpc-go v0.28.0

go.mod moves from vgi-rpc-go v0.27.0 to v0.28.0, which carries
wire-breaking transport changes of its own: __describe__ is retired in favour
of the vgi_rpc.Reflection.v1 protocol, raw transports now require the
vgi_rpc.protocol routing key at construction, and the protocol hash is
redefined. An implementation still built against v0.27.0 cannot talk to a
v0.28.0 peer, so the bump is not optional. None of those surfaces are touched
directly by this SDK — the bump needed no call-site edits here.

Performance: the FINALIZE flush is no longer carried whole in every continuation token

The headline for anyone running table-in-out functions over HTTP.

A flush is drained one batch per turn, and over HTTP every turn is a separate
request that rebuilds producer state from a continuation token.
FinalizeProducerState.BatchIPC held all N batches and was never truncated as
BatchIdx advanced — so each token carried all N (O(N²) bytes on the wire)
and rehydrateFinalize then decoded all N to emit one (O(N²) CPU). At
N=5000 that was 12.5M IPC decodes.

The flush is now written once to the worker's execution-scoped state log
(StateAppend / StateLogScan) and only a cursor rides the token; each turn
scans exactly one row. Measured:

before after
isolated 5000-row finalize query 161.0s 5.4s
multi_batch_finalize.test 493s 17.2s
whole Go HTTP integration lane ~10min ~2min

Now linear in N, and faster than the Python reference at every size.

Three properties worth knowing:

  • Byte-stream transports are untouched. The offload happens inside
    GobEncode, i.e. only when a continuation token is actually written. stdio,
    the AF_UNIX launcher and shm never reach it and keep their in-memory drain —
    which the unchanged launcher lane confirms.
  • Stateless resume is preserved. The log lives in the durable shared
    FunctionStorage backend, not process memory, so a continuation arriving at a
    worker that has never seen the stream still resumes from the token alone. A
    backend that cannot serve a state log falls back to the old inline carry —
    slower, never broken.
  • Log keys are UUID-suffixed per stream, so the several finalize substreams
    a parallel query opens under one execution_id cannot drain each other's
    batches.

Also

ci/run-integration.sh drops a stale transport-version claim in the comment
justifying the http-lane dynamic_filter.test exclusion (both halves of it were
already false). The exclusion itself stays, now labelled a removal candidate
rather than a known-good removal.

Known issue: the shm CI lane was exercised to ~79 of 331 files (#9)

Stated plainly, because anyone deploying over the shared-memory side channel
should know it before they upgrade.

The integration (shm) lane — the AF_UNIX launcher plus the POSIX shm side
channel — does not complete on this release. The prebuilt
haybarn-unittest process takes a SIGSEGV (exit 139, core dumped) at
test/sql/integration/cache/explain_stats.test, file 78 of 331, so that lane
actually ran roughly 79 of 331 files. It reproduces on re-run, at the same
file. The stdio, launch and http lanes each ran all 331 files green,
as did lint / vet / golangci-lint / build / unit tests.

This is a crash in the test runner, not a product failure, and the evidence
that it is not in this release's diff is strong:

  • it is already present at the vgi.v2 commit, before the finalize change;
  • every shm source in vgi-rpc-go is byte-identical between v0.27.0 and v0.28.0,
    so the transport bump changed no shm code;
  • the launch lane — the same lane minus the side channel — passes the same
    file;
  • the SIGSEGV is taken by the C++ process, not by the Go worker;
  • ci/run-integration.sh already excludes nested_type_combinations.test from
    every lane as "a property of that C++ build, not the worker".

The same file was also run under shm against a locally built runner, a locally
built extension at 5d45bb4 and a freshly built worker from this commit, and it
passes there. But that was macOS/arm64 against a locally built binary while
the crash is linux/amd64 against the prebuilt one — two variables differ at
once, so the prebuilt runner is the leading suspect rather than a proven cause.
The crash is narrowed, not explained.

Practical effect: if you use VGI_RPC_SHM_SIZE_BYTES together with DuckDB's VGI
result cache, that combination is currently unproven on this release. Everything
the other three transports cover is fully exercised. Details, logs and next
steps are in #9.

Full changelog: Query-farm/vgi-go@v0.24.0...v0.25.0