Skip to content

v0.29.0 — protocol 1.4.0 (splits)

Choose a tag to compare

@rustyconover rustyconover released this 24 Aug 20:20
· 65 commits to main since this release

Protocol 1.4.0 — a plan() phase for VGI

table_function_plan() splits init() into two phases: plan once with the
pushdown filters and get back named, redeemable splits; then init() per
split from any process or host.

This replaces the opaque worker-side work queue, which was correct in one
process but unsound in a distributed engine — Spark, Trino and distributed
DataFusion re-run failed tasks, and with a worker-side queue a retry reads
neither what it read before nor what it missed. A split token names the work.

Also in this release:

  • Split tokens carry a plaintext integrity header with a sealed payload,
    bound to the caller principal. A keyed worker rejects an unsealed token
    (the alg:none downgrade).
  • on_plan / on_split hooks — split redemption is a third branch in
    init, distinct from primary and secondary.
  • Format branches (format_name / format_locations / format_options)
    and the single-branch branch_filter collapse fix.
  • Client capabilities on attach.
  • Positions (start_position / end_position) ship as addressability for
    incremental reads; no engine consumes them yet.
  • Generated Arrow schemas for the inner request records, plus a Java emitter so
    vgi-java checks its codec records against the protocol.
  • Requires vgi-rpc 0.43.0 for correct nullability under Annotated.

This is a forcing change. 1.4.0 refuses any peer speaking 1.3.0, so all five
SDKs release together.