Most of tutorials just explains how to use the Airflow DockerOperator using the bare metal installation; and here we will use it with Airflow on top of Docker Compose.
- First create a container with the webservice and create the
airflowuser, as described in the official docs:
$ docker-compose up airflow-initor execute the following script below:
$ bash bin/initial_setup.sh- With this initial setup made, start the webservice and other components via
docker-compose,
$ docker build -f dags/docker_job/Dockerfile -t docker_image_task . && \
docker-compose up -dor execute the following script below that will do the same thing:
$ bash bin/start.sh- Finally when you're done with your experiment, stop all containers running the following command:
$ bash bin/stop.sh
Useful links that I used during this endeavor in importance order.
-
How can I used the DockerOperator in Airflow, of I am already running Airflow in Docker?
-
Using Docker-in-Docker for your CI or testing environment? Think twice.
-
How to mount docker socket as volume in docker container with correct group
-
Permission issue on running docker command in Python subprocess through Apache Airflow