/datafusion

Modern distributed compute platform using Apache Arrow as the memory model
rust spark sql dataframe cluster compute data arrow
  1. Rust 94.3%
  2. Shell 3.7%
  3. Dockerfile 2.0%
Switch branches/tags
Nothing to show
Find file
Clone or download

Clone with HTTPS

Use Git or checkout with SVN using the web URL.

Open in Desktop Download ZIP

Launching GitHub Desktop...

If nothing happens, download GitHub Desktop and try again.

Launching GitHub Desktop...

If nothing happens, download GitHub Desktop and try again.

Launching Xcode...

If nothing happens, download Xcode and try again.

Launching Visual Studio...

If nothing happens, download the GitHub extension for Visual Studio and try again.

@andygrove
andygrove implement one example of CAST of literal
Latest commit bb32e8a Dec 16, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci save work on docker packaging (#65) Feb 20, 2018
examples implement one example of CAST of literal Dec 16, 2018
scripts cargo fmt Dec 7, 2018
src implement one example of CAST of literal Dec 16, 2018
test/data NULL support (#162) May 26, 2018
.dockerignore working on docker packaging Feb 19, 2018
.gitignore Refactoring aggregation code (#112) Apr 22, 2018
.travis.yml increase code coverage May 8, 2018
BUILDING.md fix typo in README Apr 8, 2018
CONTRIBUTING.md update README and added CONTRIBUTING.md Feb 2, 2018
Cargo.toml Add support for AND / OR expressions Dec 15, 2018
LICENSE.TXT Add license Dec 31, 2016
README.md Projection now has correct schema Dec 15, 2018

README.md

DataFusion: Modern Distributed Compute Platform implemented in Rust

License Version Build Status Coverage Status Gitter chat

DataFusion is an attempt at building a modern distributed compute platform in Rust, using Apache Arrow as the memory model.

See my article How To Build a Modern Distributed Compute Platform to learn about the design and my motivation for building this. The TL;DR is that this project is a great way to learn about building distributed systems but there are plenty of better choices if you need something mature and supported.

Status

The original POC no longer works due to changes in Rust nightly since 11/3/18 and since then I have been contributing more code to the Apache Arrow project and decided to start implementing DataFusion from scratch based on that latest Arrow code and incorporating lessons learned from the first attempt. The original POC code is is now on the original_poc branch and supports single threaded SQL execution against Parquet and CSV files using Apache Arrow as the memory model.

The current task list:

  • Delete existing code and update the README with the new plan
  • Implement serializable logical query plan
  • Implement data source for CSV
  • Implement data source for Parquet
  • Implement query execution: Projection
  • Implement query execution: Selection
  • Implement query execution: Sort
  • Implement query execution: Aggregate
  • Implement query execution: Scalar UDFs
  • Implement query execution: Array UDFs
  • Implement query execution: StructArray
  • Implement parallel query execution (multithreaded, single process)
  • Generate query plan from SQL
  • Execute SQL against DataSource
  • Implement worker node that can receive a query plan, execute the query, and return a result in Arrow IPC format
  • Kubernetes support for spinning up worker nodes
  • Implement distributed query execution

Prerequisites

  • Rust nightly (required by parquet-rs crate)

Building DataFusion

See BUILDING.md.

Gitter

There is a Gitter channel where you can ask questions about the project or make feature suggestions too.

Contributing

Contributors are welcome! Please see CONTRIBUTING.md for details.