v3.0.0

Highlights

• Opt-in Adaptive Query Execution (AQE). Distributed planning can now sample runtime stage output and choose task counts dynamically, using a cost model and completion-based estimates. Enable it with with_distributed_dynamic_task_count(true). #432, #486, #552

• New event-driven customization API for distributed planning. Register ordered handlers for the distinct planning decisions: choose a desired task count (including asynchronously, after consulting metadata), specialize a leaf into per-task work, and route tasks to workers. Handlers can defer to later handlers and the built-in defaults, so integrations only need to implement the events they own. Worker-side plan rewrites are likewise composable and run in registration order before task execution. This replaces the former TaskEstimator and worker plan-hook APIs. #564, #567, #568, #569

• Native distributed EXPLAIN ANALYZE. Query plans now show distributed stages together with coordinator and worker metrics, making it easier to inspect execution, skew, and network costs. Distributed plan display is also more compact. #558, #527

• Broadcast joins enabled by default. CollectLeft hash joins now broadcast their build side by default. This release also fixes unsafe multi-task join shapes—including certain outer, anti, nested-loop, and cross joins—and preserves build-side FETCH/limit semantics during broadcast rewriting. #574, #583, #618

• Better local execution placement. When the coordinator is also a worker, applications can register a LocalWorkerContext so single-task dynamic stages execute locally instead of making unnecessary network hops. #559

• Correct worker metrics after stream cancellation. Worker task metrics are now collected after the execution stream is dropped, so metrics finalized during early termination—such as a downstream LIMIT stopping a stream—are included. #503

Upgrade notes

3.0.0 includes source-breaking API changes. See the 3.0.0 upgrade guide for migrations from 2.0.0.

What's Changed

• Add quickstart and engine benchmark comparison to README by @gabotechs in #497
• Pass SessionConfig to Worker::add_on_plan_hook by @geoffreyclaude in #499
• Use test assertion helpers in worker plan hook tests by @geoffreyclaude in #500
• Refactor task spawner into QueryCoordinator and StageCoordinator by @gabotechs in #479
• Improve worker resolver ergonomics by @gabotechs in #501
• Collect worker metrics after stream drop by @geoffreyclaude in #503
• Fix with_new_children in leaf nodes by @gabotechs in #505
• metrics: update plan_bytes_sent to be a bytes metric by @jayshrivastava in #506
• fix comment on DistributedLeafExec by @jayshrivastava in #507
• fix: new session properties should not error in old worker versions by @jayshrivastava in #511
• Add cost model by @gabotechs in #486
• Rearrange if statements in planner by @gabotechs in #514
• Show rewritten physical plan with --debug in local benchmarks by @gabotechs in #515
• Adaptive task count assignation by @gabotechs in #432
• Fix precision sums in cost based planner by @gabotechs in #526
• Make EXPLAIN ANALYZE-like visualization less verbose by @Tristan1900 in #527
• Finalize worker metrics on coordinator channel EOS by @gabotechs in #524
• Improve docs by @gabotechs in #535
• Update README.md by @gabotechs in #536
• Add badges to README.md by @gabotechs in #537
• Abstract worker protocol from gRPC by @gabotechs in #512
• Fix TopK compute cost estimation by @gabotechs in #546
• Update docs by @gabotechs in #549
• FIX: rustc (<1.98) segfault in LatencyMetric by @thorfour in #555
• refactor: SessionConfig.extensions instead ConfigOptions.extensions by @Rich-T-kid in #550
• Add stats estimation accuracy to benchmarks by @gabotechs in #556
• Update benchmarks in README.md and make ballista work by @gabotechs in #557
• Drop velocity-based estimation in favor of driver path completion estimation by @gabotechs in #552
• Add aqe docs by @gabotechs in #561
• Local worker allocations by @gabotechs in #559
• Move children isolator union insertion before distribution by @alexanderbianchi in #565
• Introduce distributed planning lifecycle event handlers for replacing TaskEstimator by @gabotechs in #564
• Update docs for event handlers by @gabotechs in #572
• Enable broadcast joins by default by @gabotechs in #574
• Add guidlines for AI contributions and reviews by @gabotechs in #581
• Define routing in terms of events by @gabotechs in #567
• Define worker plan rewrites in terms of events by @gabotechs in #568
• Make desired task-count handlers async by @gabotechs in #569
• Auto implement event handlers for Arcs by @gabotechs in #585
• Support distributed EXPLAIN ANALYZE by @gabotechs in #558
• docs: add 3.0.0 upgrade guide by @gabotechs in #589
• build: make the datafusion parquet and sql features optional by @philippemnoel in #588
• fix: Normalize CollectLeft Joins and similar by @barbarj in #583
• fix: preserve broadcast build-side fetch by @shinzoxD in #618
• Adjust documentation about scale_up_leaf_node partition counts. by @stuhood in #619
• Remove remote benchmarks by @gabotechs in #621
• refactor: centralize protobuf code generation by @gabotechs in #627
• refactor: move remote benchmark worker upstream by @gabotechs in #630

New Contributors

• @Tristan1900 made their first contribution in #527
• @thorfour made their first contribution in #555
• @alexanderbianchi made their first contribution in #565
• @philippemnoel made their first contribution in #588
• @barbarj made their first contribution in #583
• @shinzoxD made their first contribution in #618
• @stuhood made their first contribution in #619

Full Changelog: v2.0.0...v3.0.0