Charmander Scheduler Lab
Charmander is a lab environment for measuring and analyzing resource-scheduling algorithms.
The project got started in Summer 2014 by Theodora Chu as an internship project. It was motivated by a paper from Stanford University: "Quasar: Resource-Efficient and QoS-Aware Cluster Management".
Charmander at its core provides an easy to use environment to a) schedule and deploy compute-tasks in a multi-node setup and b) measure the corresponding cpu, memory, and network-loads.
The collected measurements can afterwards be analyzed using the included Spark analytics workbench and subsequently those results can be fed back in to the scheduler.
Obviously this lab-setup can be used for other use-cases like testing and analyzing machine-learning based anomaly-detection, noisy-neighbor detection, or profiling algorithms, or simply serving as the load-pattern verification authority in a continuous integration environment.
Setup and Run Charmander
All that is required to run a simple lab setup and an experiment is Vagrant, VirtualBox, curl, a fast internet connection, and a bit of spare time.
All the steps are automated and are part of simple scripts that come with the Charmander project. All the additional software needed during the setup gets installed and built inside the VMs, nothing additionally gets installed on the host itself.
Btw, if at any time during installation you run in to an error, please follow the steps outlined in Tips and Tricks
- Configure and build the different nodes with Vagrant and VirtualBox
- Reload and reset environment
- Clone the Charmander Scheduler projects
- Compile it inside the master node
- Deploy and run it
- Build all the Docker images for the full analytics stack on the analytics node
Run some Experiments
- List of all the different Scheduler REST APIs
- List of all the include scripts/tools
- Some tips and tricks for common issues
The different Charmander projects on GitHub
The main project, this project, it contains all the scripts to set up the lab.
Charmander Scheduler is our Framework for Mesos.
A slimmed-down version of Google's Heapster project.
This project contains Charmander-specific helper functions for Spark.
The MaxUsage-Experiment analyzes the actual memory usage of a running simulators and uses that result to overwrite the memory-allocation for subsequent run-requests for the same simulators.
The SparkKernel-Experiment is using SparkKernel to calculate max-memory usage.
Additional thanks goes to
some additional open source projects and a blog post that have inspired us: