Visualize pipeline definitions for AWS Data Pipeline
Ruby
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin
graphs
lib
spec
.cane
.gitignore
.rspec
.travis.yml
Gemfile
README.md
Rakefile
pipely.gemspec

README.md

pipely

Gem Version Build Status Code Climate

Build, deploy, and visualize pipeline definitions for AWS Data Pipeline

"AWS Data Pipeline is a web service that you can use to automate the movement and transformation of data. With AWS Data Pipeline, you can define data-driven workflows, so that tasks can be dependent on the successful completion of previous tasks."

http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/what-is-datapipeline.html

Install

(First install GraphViz if it is not already installed.)

Into Gemfile from rubygems.org:

gem 'pipely'

Into environment gems from rubygems.org:

gem install pipely

Usage

Rake Tasks

Coming soon.

rake definition        # Graphs the full pipeline definition using Graphviz
rake deploy            # Deploy pipeline
rake graph             # Graphs the full pipeline definition using Graphviz
rake upload_steps      # Upload Data Pipeline steps to S3

In order to run these tasks, you must have an aws-sdk credentials file. This can be created by running aws cli configure, as described in the aws-sdk docs.

Command-line Interface

(If you used the Gemfile install, prefix the below commands with bundle exec.)

To render a JSON pipeline definition as a PNG graph visualization:

pipely definition.json

To specify the output path for PNG files:

pipely -o path/to/graph/pngs definition.json