A small repo that creates the small infrastructure to run airflow.
This repo is designed to get the necessary components of airflow installed
and talking to each other. The compose file and the airflow directory do a
few things:
- Build an image that has airflow and its dependencies.
- Setup a redis image for airflow to use as its message broker.
- Setup a postgres image for airflow to use as its backend database.
Running
The docker-compose documentation will serve you well, but in short to get
this running:
- Build the set of images via:
docker-compose build - Run the containers via:
docker-compose up
You should then be able to go <docker-ip>:8080 and see the airflow UI.
Warning
This is a simple way to get airflow up and play with it, and because of that, the tooling has not been put in place to persist the data. I.e. if you rebuild the images you will lose your data. If you'd like to use this in production it would be behoove of you to setup a dedicated database... the good news is, if you're looking for workflow management tools, you likely already have.
Similar Project
puckel/docker-airflow is a similar project, and was used as this project's
base, however this repo uses general Dockerfiles as well as postgres
and redis as opposed to mysql and rabbit.