NOTE: The Docker system is inspired from this repo, though we have customized the dockerfile as well as its config for our custom use case.
This is the code for Apache Airflow Tutorials playlist by Tuan Vu on Youtube
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.
- Clone this repo
- Install the prerequisites
- Run the service
- Check http://localhost:8080
- Done! 🎉
- Install Docker
- Install Docker Compose
- Following the Airflow release from Python Package Index
Run the web service with docker
docker-compose up -d
# Build the image
# docker-compose up -d --build
Check http://localhost:8080/
docker-compose logs
- Displays log outputdocker-compose ps
- List containersdocker-compose down
- Stop containers
If you want to run airflow sub-commands, you can do so like this:
docker-compose run --rm webserver airflow list_dags
- List dagsdocker-compose run --rm webserver airflow test [DAG_ID] [TASK_ID] [EXECUTION_DATE]
- Test specific task
If you want to run/test python script, you can do so like this:
docker-compose run --rm webserver python /usr/local/airflow/dags/[PYTHON-FILE].py
- Test python script
If you want to use Ad hoc query, make sure you've configured connections: Go to Admin -> Connections and Edit "postgres_default" set this values:
- Host : postgres
- Schema : airflow
- Login : airflow
- Password : airflow