Performance modelling system for Distributed Stream Processing Systems (DSPS) such as Apache Heron and Apache Storm
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.github Inital commit Jun 7, 2018
api fixexception Aug 25, 2018
common fixtopologycurrentpart1 Aug 22, 2018
config fixtopologycurrentpart1 Aug 22, 2018
docs Resolved Merge with Tom's version Aug 11, 2018
graph using old zookeeper code Aug 13, 2018
metrics refactoring code to use different metrics source for spout processing… Aug 13, 2018
model removed extra print Aug 13, 2018
paths
performance_prediction modified code to match the format of data from modified cuckoo client Aug 16, 2018
tests added capacity as a metric to validator Aug 17, 2018
tools/heron removed bug in tracker stats code that leads to grouping percentages … Aug 5, 2018
traffic_provider fixpackingplan Aug 25, 2018
.gitignore add paths to ignore Aug 22, 2018
AUTHORS.md Inital commit Jun 7, 2018
CONTRIBUTING.md
LICENSE Resolved Merge with Tom's version Aug 11, 2018
Pipfile Resolved Merge with Tom's version Aug 11, 2018
Pipfile.lock Resolved Merge with Tom's version Aug 11, 2018
README.md cleaning up names Aug 14, 2018
__init__.py Resolved Merge with Tom's version Aug 11, 2018
app.py Fixed parameter type errors Jun 8, 2018
loader.py Inital commit Jun 7, 2018
logs.py Inital commit Jun 7, 2018

README.md

Caladrius

Performance modelling for Distributed Stream Processing Systems (DSPS) such as Apache Heron and Apache Storm.

Full details can be found on the documentation site.

NOTE: Caladrius is a prototype project, which is the result of a 3 month internship with Twitter's Real Time Compute Team. It should be considered alpha level software. All contributions are welcome, please see the contributing page on the documentation website for more details.

Setup

Python

Caladrius requires Python 3.6, additional Python dependencies are listed in the Pipfile. Dependencies can be installed using pipenv by running the following command in the caladrius root directory:

$ pipenv install 

Add the --dev flag to the above command to install development dependencies.

Caladrius should also be added to your PYTHONPATH. The best way to do this is by adding the folder above the Caladrius repo to the PYTHONPATH environment variable using a command like the one below:

$ export PYTHONPATH=$PYTHONPATH:<path/to/folder/above/caladrius>

This line should be added to your .profile (or similar) start up script to preserve this across reboots.

####Troubleshooting Errors such as the following entail that the PYTHONPATH is not set correctly.

File "app.py", line 15, in <module>
    from caladrius import logs

Another way to ensure that pipenv is able to read the environment variables is to create a .env file in the project directory and add the PYTHONPATH there.

Graph Database

Caladrius requires a Gremlin Server instance running TinkerPop 3.3.2 or higher.

The reference gremlin sever can be downloaded from here.

The Gremlin server should have the Gremlin Python plugin installed:

$ gremlin-server.sh install org.apache.tinkerpop gremlin-python 3.3.3

Start the server with the gremlin python config (included in the standard server distribution):

$ bin/gremlin-server.sh start conf/gremlin-server-modern-py.yaml

Please note: The default settings for the Gremlin Server result in an in-memory TinkerPop Server instance. If graphs need to be persisted to disk then these settings can be altered in the appropriate configuration file in the conf directory of the Gremlin Server distribution.

Running Caladrius

Configuration

All configuration is done via the yaml file provided to the app.py script (see section below). This file defines the models run by the various API endpoints and any connection details, modelling variables or other configurations they may require.

An example configuration file with sensible defaults is provided in config/main.yaml.example. You should copy this and edit it with your specific configurations.

Starting the API Server

The Caladrius API server can be started by running the app.py script in the root directory. This can be run in the appropriate virtual environment using pipenv (make sure your python command points to Python 3):

$ pipenv run python app.py --config /path/to/config/file

Additional command line arguments are available via:

$ pipenv run python app.py --help

Documentation

Documentation for stable releases is hosted on ReadTheDocs.

If you want to build the latest documentation then this can be done using Sphinx. Assuming you have installed the development dependencies above, the docs can be built using the following commands in the repository root:

$ pipenv run sphinx-apidoc -f -o docs/source . tests/*
$ cd docs
$ pipenv run make html

This will place the constructed html documentation in the docs/build directory.

Security

If you spot any security or other sensitive issues with the software please report them via the Twitter HackerOne bug bounty program.

Using the API

The software provides multiple endpoints for a user to find out how different packing plans will perform for a single topology. Here, we provide examples of how to call the APIs from the command-line.

Heron Current API

In this example, the WindowedWordCountTopoology has three components (spouts -> bolt -> bolt). Each operator in the job has one running task/instance only.

curl -H 'Content-Type: application/json' -d '{ "1" : {"default": 101.4}, "2": {"default": 104.3}, "3" : {"default" : 101.5}  }'  
-X POST "<Caladrius URL>:5000/model/topology/heron/current/WindowedWordCountTopology?cluster=<cluster-name>&environ=test&model=queueing_theory&source_hours=2"

Caveat: Caladrius currently does not support calculations for topologies that have only two levels. This is because a topology with two levels consists of spouts (with only outgoing streams) and sink bolts (with possibly only incoming streams). Some of Caladrius' calculations such as measuring the input to output tuple ratios cannot be applied to such operators.