docker-airflow

Informations

• Based on Python (3.6-slim) official Image python:3.6-slim and uses the official Postgres as backend and Redis as queue
• Install Docker
• Install Docker Compose
• Following the Airflow release from Python Package Index

First step: Modify configuration files

Must modify three files:

• docker-airflow/dags/airflow_code/retail_config.py
• docker-airflow/dags/airflow_code/retail_db_connection.py
• docker-airflow/config/airflow.cfg

second step： Installation

Executive order.

./run.sh

UI Links

• Airflow: localhost:6688
• Flower: localhost:5555

If you get an error like：Broken DAG: [/usr/local/airflow/dags/airflow_code/retail_workflow_dev.py] 'Variable sql_path does not exist',carried out the dag:retail_db_connection

Username Password

• Username:admin
• Password:lz12345+

If you want to desgin by yourself,you can modify the file:docker-airflow/config/add_user.py,then redo './run.sh'

manual add user

docker-compose -f docker-compose-CeleryExecutor.yml exec 'webserver' /entrypoint.sh python /usr/local/airflow/add_user.py

Usage

By default, docker-airflow runs Airflow with SequentialExecutor :

docker run -d -p 8080:8080 puckel/docker-airflow webserver

If you want to run another executor, use the other docker-compose.yml files provided in this repository.

For LocalExecutor :

docker-compose -f docker-compose-LocalExecutor.yml up -d

For CeleryExecutor :

docker-compose -f docker-compose-CeleryExecutor.yml up -d

NB : If you want to have DAGs example loaded (default=False), you've to set the following environment variable :

LOAD_EX=n

docker run -d -p 8080:8080 -e LOAD_EX=y puckel/docker-airflow

If you want to use Ad hoc query, make sure you've configured connections: Go to Admin -> Connections and Edit "postgres_default" set this values (equivalent to values in airflow.cfg/docker-compose*.yml) :

• Host : postgres
• Schema : airflow
• Login : airflow
• Password : airflow

For encrypted connection passwords (in Local or Celery Executor), you must have the same fernet_key. By default docker-airflow generates the fernet_key at startup, you have to set an environment variable in the docker-compose (ie: docker-compose-LocalExecutor.yml) file to set the same key accross containers. To generate a fernet_key :

docker run puckel/docker-airflow python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)"

Configurating Airflow

It's possible to set any configuration value for Airflow from environment variables, which are used over values from the airflow.cfg.

The general rule is the environment variable should be named AIRFLOW__<section>__<key>, for example AIRFLOW__CORE__SQL_ALCHEMY_CONN sets the sql_alchemy_conn config option in the [core] section.

Check out the Airflow documentation for more details

You can also define connections via environment variables by prefixing them with AIRFLOW_CONN_ - for example AIRFLOW_CONN_POSTGRES_MASTER=postgres://user:password@localhost:5432/master for a connection called "postgres_master". The value is parsed as a URI. This will work for hooks etc, but won't show up in the "Ad-hoc Query" section unless an (empty) connection is also created in the DB

Custom Airflow plugins

Airflow allows for custom user-created plugins which are typically found in ${AIRFLOW_HOME}/plugins folder. Documentation on plugins can be found here

In order to incorporate plugins into your docker container

• Create the plugins folders plugins/ with your custom plugins.
• Mount the folder as a volume by doing either of the following:
  • Include the folder as a volume in command-line -v $(pwd)/plugins/:/usr/local/airflow/plugins
  • Use docker-compose-LocalExecutor.yml or docker-compose-CeleryExecutor.yml which contains support for adding the plugins folder as a volume

Install custom python package

• Create a file "requirements.txt" with the desired python modules
• Mount this file as a volume -v $(pwd)/requirements.txt:/requirements.txt (or add it as a volume in docker-compose file)
• The entrypoint.sh script execute the pip install command (with --user option)

Wanna help?

Fork, improve and PR. ;-)