Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


This PR does a generic pass over the docs:
- Adds cross-reference links where appropriate.
- Makes code blocks copyable.
- Minor grammar fixes/prose tweaks.

Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Daft dataframes can load any data such as PDF documents, images, protobufs, csv, parquet and audio files into a table dataframe structure for easy querying

Github Actions tests PyPI latest tag Coverage slack community

WebsiteDocsInstallation10-minute tour of DaftCommunity and Support

Daft: the distributed Python dataframe for complex data

Daft is a fast, Pythonic and scalable open-source dataframe library built for Python and Machine Learning workloads.

Daft is currently in its Beta release phase - please expect bugs and rapid improvements to the project. We welcome user feedback/feature requests in our Discussions forums

Table of Contents

About Daft

The Daft dataframe is a table of data with rows and columns. Columns can contain any Python objects, which allows Daft to support rich complex data types such as images, audio, video and more.

  1. Any Data: Columns can contain any Python objects, which means that the Python libraries you already use for running machine learning or custom data processing will work natively with Daft!
  2. Notebook Computing: Daft is built for the interactive developer experience on a notebook - intelligent caching/query optimizations accelerates your experimentation and data exploration.
  3. Distributed Computing: Rich complex formats such as images can quickly outgrow your local laptop's computational resources - Daft integrates natively with Ray for running dataframes on large clusters of machines with thousands of CPUs/GPUs.

Getting Started


Install Daft with pip install getdaft.

For more advanced installations (e.g. installing from source or with extra dependencies such as Ray and AWS utilities), please see our Installation Guide


Check out our 10-minute quickstart!

In this example, we load images from an AWS S3 bucket and run a simple function to generate thumbnails for each image:

import daft as daft

import io
from PIL import Image

def get_thumbnail(img: Image.Image) -> Image.Image:
    """Simple function to make an image thumbnail"""
    imgcopy = img.copy()
    imgcopy.thumbnail((48, 48))
    return imgcopy

# Load a dataframe from files in an S3 bucket
df = daft.from_glob_path("s3://daft-public-data/laion-sample-images/*")

# Get the AWS S3 url of each image
df =["path"].alias("s3_url"))

# Download images and load as a PIL Image object
df = df.with_column("image", df["s3_url"] data:, return_dtype=daft.DataType.python()))

# Generate thumbnails from images
df = df.with_column("thumbnail", df["image"].apply(get_thumbnail, return_dtype=daft.DataType.python()))

Dataframe code to load a folder of images from AWS S3 and create thumbnails


Benchmarks for SF100 TPCH

To see the full benchmarks, detailed setup, and logs, check out our benchmarking page.

More Resources

  • 10-minute tour of Daft - learn more about Daft's full range of capabilities including dataloading from URLs, joins, user-defined functions (UDF), groupby, aggregations and more.
  • User Guide - take a deep-dive into each topic within Daft
  • API Reference - API reference for public classes/functions of Daft


To start contributing to Daft, please read


To help improve Daft, we collect non-identifiable data.

To disable this behavior, set the following environment variable: DAFT_ANALYTICS_ENABLED=0

The data that we collect is:

  1. Non-identifiable: events are keyed by a session ID which is generated on import of Daft
  2. Metadata-only: we do not collect any of our users’ proprietary code or data
  3. For development only: we do not buy or sell any user data

Please see our documentation for more details.

Related Projects

Dataframe Query Optimizer Complex Types Distributed Arrow Backed Vectorized Execution Engine Out-of-core
Daft Yes Yes Yes Yes Yes Yes
Pandas No Python object No optional >= 2.0 Some(Numpy) No
Polars Yes Python object No Yes Yes Yes
Modin Eagar Python object Yes No Some(Pandas) Yes
Pyspark Yes No Yes Pandas UDF/IO Pandas UDF Yes
Dask DF No Python object Yes No Some(Pandas) Yes

Check out our dataframe comparison page for more details!


Daft has an Apache 2.0 license - please see the LICENSE file.