airflow-docker

This repository contains Dockerfile of apache-airflow for Docker's automated build published to the public Docker Hub Registry.

Informations

• Based on Python (3.6-stretch) official Image python:3.6-stretch
• Install Docker
• Install Docker Compose
• Following the Airflow release from Python Package Index

Installation

Pull the image from the Docker repository.

    docker pull shinyorke/airflow

Build

    docker build --rm -t shinyorke/airflow .

Settings

environments for airflow.cfg

sample:env_example

# User & HOME
AIRFLOW_USER=airflow
AIRFLOW_HOME=/usr/local/airflow

# Database(MySQL)
AIRFLOW_DB_CONN=mysql://airflow:password@localhost:3306/airflow

# Database(sqlite)
# AIRFLOW_DB_CONN=sqlite:////usr/local/airflow/airflow.db

# Airflow webserver settings
AIRFLOW_WEB_PROTOCOL=http
AIRFLOW_WEB_HOST=localhost
AIRFLOW_WEB_PORT=8080
AIRFLOW_WEB_WORKERS=4

# Airflow worker settings
AIRFLOW_WORKER=redis
AIRFLOW_WORKER_HOST=redis
AIRFLOW_WORKER_PORT=6379
AIRFLOW_WORKER_DATABASE=0

# Airflow base settings
# SequentialExecutor, LocalExecutor, CeleryExecutor
AIRFLOW_EXECUTOR=SequentialExecutor
AIRFLOW_EXAMPLES=False
AIRFLOW_AUTH=False
SQL_ALCHEMY_POOL_SIZE=5
SQL_ALCHEMY_POOL_RECYCLE=3600

# Application PATH
PYTHONPATH=/usr/local/airflow

Usage

webserver

    docker run -p 8080:8080 --env-file=./env_example shinyorke/airflow webserver init

scheduler

    docker run --env-file=./env_example shinyorke/airflow scheduler init

worker

    docker run --env-file=./env_example shinyorke/airflow worker init

Sample(docker-compose)

airflow service example

• webserver
• scheduler
• worker
• backend

  • MySQL(airflow db)

  • redis(workder queue)

    docker-compose -f docker-compose-example-db.yml up -d
    docker-compose -f docker-compose-example.yml build
    docker-compose -f docker-compose-example.yml up -d
    

Wanna help?

Fork, improve and PR. Sorry m(_ _)m

Maintainer

Shinichi Nakagawa

• Github
• @shinyorke(Twitter)
• Facebook