Twitter Ambrose is a platform for visualization and real-time monitoring of MapReduce data workflows. It presents a global view of all the map-reduce jobs derived from your workflow after planning and optimization. As jobs are submitted for execution on your Hadoop cluster, Ambrose updates its visualization to reflect the latest job status, polled from your process.
Ambrose provides the following in a web UI:
- A chord diagram to visualize job dependencies and current state
- A table view of all the associated jobs, along with their current state
- A highlight view of the currently running jobs
- An overall script progress bar
Ambrose is built using the following front-end technologies:
Ambrose is designed to support any Hadoop workflow runtime, but current support is limited to Apache Pig.
Follow @Ambrose on Twitter to stay in touch!
- Pig - See pig/README.md
- Cascading - future work
- Scalding - future work
- Cascalog - future work
- Hive - future work
Below is a screenshot of the Ambrose UI. Each arc segment on the circle represents a map-reduce job. Dependencies between jobs are represented by chords which connect job arc segments. Grey jobs have not yet run, bright green jobs are running and light green jobs are completed. When the mouse hovers over a job, its arc and input dependencies are highlighted blue. Clicking on the job will select it, updating the contents of the table to the right of the diagram with information about the selected job.
Note that Each job arc is bisected; Chords on one half of the arc connect to predecessor jobs while chords on the other half connect to successor jobs. For example, in the diagram below jobs 1 and 3 have no predecessors while jobs 16, 20, 21, and 22 have no successors (their outputs represent the final result of this workflow).
Note that the chord diagram shown is our first pass at visualizing the workflow, and there's room for improvement. We'd like to support other visualizations as well, like a graph of the workflow DAG. If you develop an improved visualization, be sure to send us a pull request!
To get started with Ambrose, first clone the Ambrose Github repository:
git clone https://github.com/twitter/ambrose.git cd ambrose
Next, you can try running the Ambrose demo on your local machine. The
ambrose-demo script starts a
local instance of the Ambrose app server with sample data. Start the demo Abrose server with the
following command and then browse to
Finally, you can run Ambrose with an actual Pig script. To do so, you'll need to build the Ambrose distribution and untar it:
./bin/ambrose-package VERSION=0.1.0-SNAPSHOT tar zxvf ambrose-$VERSION.tar.gz
You can then run the following commands to execute
path/to/my/script.pig with an Ambrose app server
embedded in the Pig client:
cd ambrose-$VERSION ./bin/pig-ambrose -f path/to/my/script.pig
Now, browse to http://localhost:8080/web/ to see the progress of your script
using the Ambrose UI. To override the default port, export
AMBROSE_PORT before invoking
An initial release will be pushed to Maven shortly.
How to contribute
Bug fixes, features, and documentation improvements are welcome! Please fork the project and send us a pull request on Github. You can submit issues on Github as well.
Here are some high-level goals we'd love to see contributions for:
- Improve the front-end client
- Add other visualization options, like a DAG view
- Create a new back-end for a different runtime environment
- Create a standalone Ambrose server that's not embedded in the workflow client
For transparency and insight into our release cycle, releases will be numbered with the follow format:
And constructed with the following guidelines:
- Breaking backwards compatibility bumps the major
- New additions without breaking backwards compatibility bumps the minor
- Bug fixes and misc changes bump the patch
For more information on semantic versioning, please visit http://semver.org/.
Copyright 2012 Twitter, Inc.
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0