Skip to content
A deployment framework for Austin Transportation's ETL tasks
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.


This repository houses a deployment framework for Austin Transportation's ETL scripting tasks. It uses Python, Docker and cron to schedule and monitor each task. The framework's core components are:

Builder (

The builder generates a docker run command for each task. It handles the syntax for passing tasks to the launcher and packages each command as a shell script which can be installed as a cron job.

Deployer (

The deployer installs each task as a cron job on a Linux host.

Launcher (

The launcher acts as a wrapper for each ETL script and manages logging, email notifications, and job registration for incremental data loading.


  • a Linux host with Python v2.7+ installed

  • PyYAML


  1. Clone this repo on a Linux host: git clone

  2. Define script and docker run parameters in config/scripts.yml and config/docker.yml respectively.

  3. $ bash to generate shell scripts and cron entries.

  4. $ bash to install crontab on host.


As a work of the City of Austin, this project is in the public domain within the United States.

Additionally, we waive copyright and related rights of the work worldwide through the CC0 1.0 Universal public domain dedication.

You can’t perform that action at this time.