Call a VGI worker over HTTP from Java — and worker logs now survive an HTTP stream.
Server-side fix — worth upgrading for on its own
A worker serving streams over HTTP has been dropping every log line after the first turn. HttpStreamHandler.runExchange built an OutputCollectorSink and never bound it to a writer, so each continuation's ctx.clientLog(...) buffered into a list that died with the request. RpcServer.serveStream binds its sink once for the whole tick loop because a pipe has one output stream; HTTP has one response per turn. Identical worker code therefore logged correctly over a pipe and silently lost everything over HTTP — worst on long streams, where progress logs are the only in-band diagnostic a caller has.
The error branch had the same defect: a turn that logged and then threw answered with a bare error batch, discarding exactly the lines that say where it broke. Errors now carry the turn's logs, in the order a client reads them.
This affects existing deployments. No client change is needed to benefit.
New: HttpRpcConnection
There was previously no way to call a worker over HTTP from Java — the http package was entirely server-side, and all four RpcTransport implementations are byte-stream oriented, which HTTP is not. This is a sibling of RpcConnection, not another transport, and offers the same proxy(Class) surface so call sites don't change when the transport does.
try (HttpRpcConnection conn = HttpRpcConnection.builder("http://host:8080/vgi")
.bearerToken(token)
.build()) {
MyService svc = conn.proxy(MyService.class);
long n = svc.add(2, 3); // unary
try (RpcStream<?> s = svc.scan(1000)) { // producer
for (AnnotatedBatch b : s.batches()) { ... }
}
}Unary, producer and exchange streams, @StreamHeader, in-band logs, bearer auth, and errors surfacing as RpcError. Scope is identity encoding only — no compression negotiation, OAuth, sticky sessions, upload-URL fallback, or shared memory. A response-side externalized batch fails loudly rather than arriving empty.
Two client-side bugs were fixed on the way: a zero-row exchange answer was swallowed as a cursor marker (a producer's continuation marker and an exchange's cursor-carrying answer are byte-identical, so only the call shape can disambiguate them), and cancel() was a silent no-op before any cursor had been read — a producer's token trails its data, so cancelling early left worker state to expire on TTL, on precisely the long streams worth abandoning.
Known gaps for production parity
Compression negotiation is the significant one: a worker advertises its codec set on VGI-Supported-Encodings and this client pins identity, which costs real bandwidth on a remote link for large Arrow bodies. Also deferred: 413 → externalize-and-retry, response-side external-location resolution, transient-failure retry, sticky sessions, and OAuth integration.