Declarative Differential Dataflows
This is currently a research project, not feature-complete in any way, and overall not ready for use in production services.
This project provides dynamic synthetization of differential dataflows at runtime. Included in this repository is the library itself, as well as a server accepting query plans serialized as JSON, via CLI, TCP, or WebSocket connections. A parser generating such plans from a Datalog-like query language is provided in Clojure(Script), but other query grammars or parsers from other host languages should be able to utilize the same query plan interface.
Build / Run
Assuming an up-to-date Rust environment, the server can be built and run via
cargo run --release --bin server -- <timely args> -- <server args>
The server executable accepts two sets of arguments separated by
one for configuring timely dataflow
and the other for configuring the server.
OPTION | DESCRIPTION | DEFAULT --port | port to listen at | 6262 --enable-cli | accept commands via stdin? | false --enable-history | keep full traces | false
Logging at a specific level can be enabled by setting the
environment variable to
Read the high-level motivation for this project.
Important architectural decisions are documented in the docs/adr/ sub-directory.
Documentation for this package can be built via
cargo doc --no-deps
and viewed in a browser via
cargo doc --no-deps --open. Please refer
declarative_dataflow::plan::Plan for documentation on the
available operators. The tests/ directory contains usage
Query plans are rather cumbersome to write manually and do not map to any interesting, higher-level semantics. Currently we provide a Datalog front end written in Clojure.