Build, deploy, and visualize pipeline definitions for AWS Data Pipeline
"AWS Data Pipeline is a web service that you can use to automate the movement and transformation of data. With AWS Data Pipeline, you can define data-driven workflows, so that tasks can be dependent on the successful completion of previous tasks."
(First install GraphViz if it is not already installed.)
Into Gemfile from rubygems.org:
Into environment gems from rubygems.org:
gem install pipely
rake definition # Graphs the full pipeline definition using Graphviz rake deploy # Deploy pipeline rake graph # Graphs the full pipeline definition using Graphviz rake upload_steps # Upload Data Pipeline steps to S3
In order to run these tasks, you must have an aws-sdk credentials file.
This can be created by running
aws cli configure, as described in the aws-sdk docs.
(If you used the Gemfile install, prefix the below commands with
To render a JSON pipeline definition as a PNG graph visualization:
To specify the output path for PNG files:
pipely -o path/to/graph/pngs definition.json