Skip to content

Latest commit

 

History

History

dagster-integration

Folders and files

NameName
Last commit message
Last commit date

parent directory

..

assets

assets

 
 

data

data

 
 

jobs

jobs

 
 

.gitignore

.gitignore

 
 

Dagster Demo Existing DAG.ipynb

Dagster Demo Existing DAG.ipynb

 
 

Dagster Demo New DAG.ipynb

Dagster Demo New DAG.ipynb

 
 

Dockerfile

Dockerfile

 
 

README.md

README.md

 
 

docker-compose.yml

docker-compose.yml

 
 

lakefs_test.csv

lakefs_test.csv

 
 

lakefs_test_latest_file.csv

lakefs_test_latest_file.csv

 
 

workspace.yaml

workspace.yaml

 
 

README.md

Integration of lakeFS with Dagster

Prerequisites

  • Docker installed on your local machine

Setup

  1. Start by cloning this repository:

    git clone https://github.com/treeverse/lakeFS-samples && cd lakeFS-samples/01_standalone_examples/dagster-integration

  2. Run the following to provision the stack which includes Python, Spark, Jupyter Notebook, JDK, Hadoop binaries and lakeFS Python client

    docker compose up

    Alternatively, if you want to provision a lakeFS server as well as MinIO for your object store, plus Jupyter then bring up the full stack:

    docker compose --profile local-lakefs up

  3. Open JupyterLab UI http://127.0.0.1:28888/ in your web browser.

Demo Instructions

Once you have successfully completed setup then open one of the provided notebooks from JupyterLab UI:

  • Dagster Demo Existing DAG
  • Dagster Demo New DAG