SnailTrail implementation
Branch: master
Clone or download
antiguru pag-construction: Group summaries by crossing snapshot boundary
Signed-off-by: Moritz Hoffmann <moritz.hoffmann@inf.ethz.ch>
Latest commit ab6dc11 Dec 15, 2017
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
logformat SnailTrail public release, initial import Dec 13, 2017
pag-construction
snailtrail SnailTrail public release, initial import Dec 13, 2017
spark-parser
tensorflow SnailTrail public release, initial import Dec 13, 2017
.gitignore SnailTrail public release, initial import Dec 13, 2017
.travis.yml SnailTrail public release, initial import Dec 13, 2017
AUTHORS SnailTrail public release, initial import Dec 13, 2017
Cargo.toml spark-parser: Insert waiting and scheduling driver activities Dec 14, 2017
LICENSE-APACHE
LICENSE-MIT SnailTrail public release, initial import Dec 13, 2017
README.md Spelling Dec 15, 2017

README.md

SnailTrail: Generalizing Critical Paths for Online Analysis of Distributed Dataflows

This project contains of several components which build on top of one another. The following list is ordered from basic libraries with supporting functionality to full applications:

Crate Description
logformat Definitions of core data types and serialization of traces (in Rust, Java).
spark-parser Preprocesses Spark driver and executor logs and re-writes timings into our intermediate representation.
pag-construction Constructs the Program Activity Graph (PAG) from a flat stream of events which denote the start/end of computation and communication. Also has scripts to generate various plots.
snailtrail Calculates a ranking for PAG edges by computing how many times an edge appears in the set of all-pairs shortest paths.

Building

You will need the standard development tools (C compiler, version control) and we assume a Unix-like system.

  1. Install Rust following the usual instructions. Make sure to install at least Rust 1.22 (stable) or newer due because we need support for custom derive (#35900) for serialization with Abomonation.

  2. Compile the code (dependencies will be fetched automatically):

    $ cargo build --release --all
  3. Run the trace visualization tool which allows you to inspect the traces generated by Timely/Flink/Spark instrumentation before constructing the PAG. This translates our MessagePack log files into the Google Chrome trace visualizer format:

    # Convert a sample raw log into our trace format
    $ cd spark-parser
    $ cargo run resources/app-20170324182509-0000
    # Second translation step into Chrome's tracing format
    $ cd logformat/rust
    $ cargo run --bin chromeviz -- path/to/trace.msgpack

    This will create a JSON file, which you can load in Google Chrome/Chromium by opening the URL chrome://tracing.

  4. Compute CP summaries using the tools in pag-construction:

    $ MODE=run ./run_all_the_things.sh
    $ MODE=summary ./run_all_the_things.sh
  5. Plot the CP summaries using the tools in pag-construction:

    $ MODE=summary ./run_all_the_things.sh

    Note that MODE=summary is the default configuration, so it can be left out.

Supported systems

At the moment we support the following systems:

System Notes
Flink Requires custom pre-processing, to be released
Spark spark-parser is required to generate trace
Timely Requires custom pre-processing, to be published
Tensorflow Tooling included
Heron Outputs traces, to be published

License

SnailTrail is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), with portions covered by various BSD-like licenses.

See LICENSE-APACHE, and LICENSE-MIT for details.