Dynamically generate Apache Airflow DAGs from YAML configuration files
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
dagfactory bump version and update changelog Nov 28, 2018
examples add support for timezone aware DAGs Nov 28, 2018
img initial commit Nov 21, 2018
scripts initial commit Nov 21, 2018
tests utils: handle invalid timezones Nov 28, 2018
.dockerignore initial commit Nov 21, 2018
.gitignore use pipenv Nov 21, 2018
.travis.yml add pytest-cov Nov 21, 2018
CHANGELOG.md bump version and update changelog Nov 28, 2018
Dockerfile Dockerfile: use airflow 1.10.1 Nov 28, 2018
LICENSE initial commit Nov 21, 2018
MANIFEST.in use pipenv Nov 21, 2018
Makefile add pytest-cov Nov 21, 2018
Pipfile add pytest-cov Nov 21, 2018
Pipfile.lock add pytest-cov Nov 21, 2018
README.md README.md: add pypi badge Nov 21, 2018
setup.py add support for timezone aware DAGs Nov 28, 2018

README.md

dag-factory

Travis CI Coverage PyPi Code Style

dag-factory is a library for dynamically generating Apache Airflow DAGs from YAML configuration files.

Installation

To install dag-factory run pip install dag-factory. It requires Python 3.6.0+ and Apache Airflow 1.9+.

Usage

After installing dag-factory in your Airflow environment, there are two steps to creating DAGs. First, we need to create a YAML configuration file. For example:

example_dag1:
  default_args:
    owner: 'example_owner'
    start_date: 2018-01-01
  schedule_interval: '0 3 * * *'
  description: 'this is an example dag!'
  tasks:
    task_1:
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: 'echo 1'
    task_2:
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: 'echo 2'
      dependencies: [task_1]
    task_3:
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: 'echo 3'
      dependencies: [task_1]

Then in the DAGs folder in your Airflow environment you need to create a python file like this:

from airflow import DAG
import dagfactory

dag_factory = dagfactory.DagFactory("/path/to/dags/config_file.yml")

dag_factory.generate_dags(globals())

And this DAG will be generated and ready to run in Airflow!

screenshot

Benefits

  • Construct DAGs without knowing Python
  • Construct DAGs without learning Airflow primitives
  • Avoid duplicative code
  • Everyone loves YAML! ;)

Contributing

Contributions are welcome! Just submit a Pull Request or Github Issue.