0.1.0

Vane 0.1.0

Vane Data is a high-performance, multimodal-native data engine for AI workloads. Built on a fork of DuckDB, it extends the core execution engine with native multimodal processing and a unified framework for local and distributed execution.

Core Features

• Distributed DuckDB execution engine
  Extends DuckDB with distributed physical plans, distributed Plan Fragments, FTE (Fault-Tolerant Execution) scheduling, and distributed operator execution, with Arrow Flight providing cross-worker Exchange/Shuffle data transport.

• Python UDFs
  Relation UDFs support row-wise map, Arrow Table-based map_batches, and one-to-many flat_map. Expression UDFs provide @vane.func, @vane.cls, and their corresponding .batch forms. Scalar, batch, and class UDFs can all be registered as SQL functions through vane.attach_function().

• AI Functions
  Provides typed Prompt and Embed APIs across the Python Expression API, Relation API, and SQL. Prompt integrates with OpenAI, Anthropic, Google, and the native vLLM backend, while Embed supports OpenAI, Google, and SentenceTransformers. Structured outputs and image Prompt inputs are available where supported by the provider.

• Native vLLM batch execution
  Implements a native PhysicalVLLM operator and Actor Pool, with bounded task submission enforced through in-flight limits. Prompts are bucketed by shared prefixes and routed to actors using prefix-aware routing to improve opportunities for reusing the vLLM Prefix Cache.
  The native vLLM Prompt path currently supports text input only.

• Adaptive multimodal batching and backpressure
  The UDF and vLLM execution paths dynamically split or combine batches according to row count, data size, and in-flight limits. Resource admission control and object-stream backpressure limit the number of queued tasks and their memory consumption.

• Fault-Tolerant Execution
  Supports task retries, Worker failure detection and replacement, Split reassignment, Attempt Fencing, cancellation, and resource cleanup.

• Ray Runner and Local Runner
  The same SQL and Relation plan model can run through either the distributed Ray Runner or the local In-Process FTE Runner. Ray Runner is the default execution path and supports both single-machine and distributed execution. Local Runner targets lightweight, lower-overhead local execution without Ray.
  Local Runner is currently experimental.

• Multimodal benchmarks
  Provides comparable Vane, Ray Data, and Daft pipelines covering audio transcription, document embedding, image classification, and video object detection.
  The current benchmarks use local files on a single-GPU machine. They represent a single-node environment and are not a direct reproduction of the original distributed Ray Data benchmark.

Contributor Acknowledgements

Thank you to all authors and co-authors whose non-automated contributions supported this release across Vane, vane-website, and demo-scene:

@kaka11chen, @QuakeWang, @caomaocao, @hubgeter, @liwuhen, @pollychen-lab, @figurant, @zy-kkk, @suxiaogang223, @jingdaws, @freemandealer, @liujiwen-up, and @StanleyXu512.