Apache Airflow is an excellent orchestrator for bath worflows. This image also comes with Apache Hive installed, to allow for queries to be remotelly performed against the Hive-Server Docker service.
Volumes can fulfill several roles in this image:
- Load DAGs for interactive editing
- Load Plugins for interactive editing
- Load credentials
- Load custom configurations
If you want to use the default DAGs and configurations in your image (which can be useful when testing a release build image, for example), comment out the volumes section in docker-compose.yml.
There are several configurations that can be automatically loaded when the container starts by overriding the pertinent files and mounting them to a volume.
There are some default connections to the other services in this project defined in this json.
You can override the extra_connections.json file to add your custom connections.
These two files will be merged at startup and the connections will be automatically created by the entrypoint script.
Airflow variables are loaded from the ./variables.json file. Override the file with your custom variables, which will be automatically loaded when the container starts up.
Airflow pools are loaded from the ./pools.json file. Override the file with your custom pools, which will be automatically loaded when the container starts up.
By default, you can find the UI at http://localhost:8080/
It's recommended that you override this file (see this section) with one that includes your Airflow's dependencies, to avoid having to install them each time you start the container up.