Dashtool is a Lakehouse build tool that builds Iceberg tables from declarative SQL statements and generates Kubernetes workflows to keep these tables up-to-date. It handles Ingestion, Transformation and Orchestration.
- Uses declarative SQL select statements as input
- git inspired data version control
- Interoperability through Apache Iceberg Table format
- Data ingestion through Singer.io
- Data processing based on Datafusion
- Workflow orchestration in Kubernetes through Argo Workflows
Dashtool constructs a DAG by analyzing all .sql files in a directory structure and creates an Iceberg Materialized View for every file.
Each file contains a SELECT statement for the Materialized View definition.
Additionally, dashtool can use the DAG to create an Argo Workflow that refreshs the Materialized Views.
During the workflow execution Argo starts Docker containers that run Datafusion to perform the refresh operation.
Dashtool goes through a build, a workflow and an apply step to turn the declarative input files into an automatically refreshing data pipeline. This is shown in the following diagram:
graph LR
git[Files in
git repo]
catalog[Iceberg tables
in catalog]
workflows[Argo
workflow]
data[Data]
git -->|dashtool build|catalog
catalog -->|dashtool workflow|workflows
workflows -->|dashtool apply|data
Check out the Documentation for a detailed description.
The build command analyzes all .sql files in the subdirectories of the current directory and creates the corresponding Iceberg Materialized Views in the catalog.
dashtool buildThe workflow command creates a lineage DAG based on the .sql files and constructs an Argo workflow based on it. It stores the Workflow configuration file in argo/workflow.yaml.
dashtool workflowTo apply the latest version of the workflow to the Kubernetes cluster run the following command:
dashtool applybrew tap dashbook/dashtool
brew install dashtoolcargo install dashtool