Starting Out

Generating airflow.cfg

You can generate an airflow.cfg file like in config/airflow.cfg by creating a container from an Airflow image and running the airflow config command inside the container.

host@machine:~/airflow$ docker run -it apache/airflow:1.10.15-python3.8 bash
airflow@63e37316975c:/opt/airflow$ airflow config
airflow@63e37316975c:/opt/airflow$ exit

Then you can copy this file over to your file system using the docker cp command.

Changes to airflow.cfg

I made some minor changes to this config file before I used it in my image. By default Airflow uses the SequentialExecutor, which allows no parallelism. There are plenty of other choices but I go for the LocalExecutor.

[core]
executor = LocalExecutor

By default, Airflow uses SQLite, although in this project, I use Postgres. I change the connection string in core.sql_alchemy_conn to the appropriate one for Postgres as set inside the postgres service definition in docker-compose.yml.

[core]
sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@postgres:5432/airflow

Secrets

The fernet_key, which is used to encrypt connection passwords in the database, starts with some randomly generated initial value. The same is true for the secret_key, which is a key used by the Flask app. Since we want to keep track of airflow.cfg's version history, it's good practice not to keep this information in plaintext in the file for security reasons. I keep these variables inside a .env file, which is used by the docker-compose.yml file.

.env

AIRFLOW__CORE__FERNET_KEY="some key"
FERNET_KEY="some key"
AIRFLOW__WEBSERVER__SECRET_KEY="some other key"
SECRET_KEY="some other key"

The airflow.cfg file can point to these variables like so:

[core]
fernet_key = $FERNET_KEY

[webserver]
secret_key = $SECRET_KEY