Daft 2026 Roadmap

[everettVT]

This roadmap outlines the big picture of what the Daft team plans to work on in the coming year, as well as some of the features to expect from these.

Please note that items on this roadmap are subject to change at any time. If there are features you would like to implement, we welcome and encourage open-source contributions. Our team is happy to provide guidance, help scope the work, and review PRs.

You can find a complete list of all Daft roadmaps here: https://github.com/Eventual-Inc/Daft/discussions/categories/roadmaps

Performance

• Performant 100TB shuffles — native peer-to-peer shuffle service (Arrow Flight RPC) in Flotilla
• Distributed engine optimization — dynamic task scheduling, reduced worker setup overhead, and dynamic partitioning in Flotilla
• Memory management — tracking and managing memory in the local runner to reduce OOMs and inform backpressure and dynamic batching
• Hash-based operator improvements — better partitioning and hash-table construction for groupby aggregations and hash joins in low-cardinality cases

Key Features

• Checkpointing — ability to checkpoint and resume long-running workloads
• Point-in-time (asof) joins — native distributed support for ML feature engineering workflows
• Kubernetes support — running distributed Daft natively on Kubernetes
• Multiple distributed backends — generic distributed backend interface with continued Ray support

Observability

• Daft Dashboard — cluster, task, and partition-level observability for Flotilla
• Improved debugging — capturing and retrieving run history and debug information
• Memory and CPU observability — resource utilization metrics capture and retrieval

Extensibility

• Arrow2 deprecation
• Native extensions — Daft extensions via Arrow C interfaces for cross-language interop
• Data source refactor — simplified source pipeline for adding native Rust sources

Please leverage linked discussions for questions on specific items. This page is intended for broader questions about the roadmap or questions about items without links. Please see this doc for the official announcement.

[chenghuichen]

Hi @everettVT — excited to see native Kubernetes support on the roadmap. I've been sketching out a design for this and put together a discussion here: #6639. Would love to get some feedback from you or anyone on the community if you have bandwidth.