The builder generates a
docker run command for each task. It handles the syntax for passing tasks to the launcher and packages each command as a shell script which can be installed as a cron job.
The deployer installs each task as a cron job on a Linux host.
The launcher acts as a wrapper for each ETL script and manages logging, email notifications, and job registration for incremental data loading.
a Linux host with Python v2.7+ installed
Clone this repo on a Linux host:
git clone https://github.com/cityofaustin/transportation-data-deploy.
Define script and
docker runparameters in
$ bash build.shto generate shell scripts and cron entries.
$ bash deploy.shto install crontab on host.
As a work of the City of Austin, this project is in the public domain within the United States.
Additionally, we waive copyright and related rights of the work worldwide through the CC0 1.0 Universal public domain dedication.