A proof of concept application for distributed image processing in the cloud. The application was created for the 2012 Cloud Computing course of the Delft University of Technology.
Note: all code found in this repository is highly experimental and in prototype shape.
What is this?
This is an application used for evaluating the design of a cloud computing system for a course at the Delft University of Technology. It is able to autonomously start and stop virtual machines as the load of the system varies. It combines a Symfony2 PHP application together with:
The high level overview of the system can be found below.
All in all the system does not actually process any images, instead it primarily keeps your CPU busy while drawing some graphs. The image processing is simulated by letting the workers of the system sleep for a certain amount of time. More information on the application can be found in the accompanying report.
What to do now?
If you enjoy the graph above you can go on and actually setup the system (recommended ;) ) with the instructions below. An alternative would be watching a video of our primary runs: https://vimeo.com/53266455.
In order to run the system Vagrant and PHP have to be installed on the host computer.
All console commands are assumed to be run from the root level of the project.
Start by cloning the repository. After cloning initialize the submodules:
$ git submodule update --init
The dependencies of the application can be installing using composer:
$ curl https://getcomposer.org/installer | php $ composer.phar install --dev
Although most of the application is automated, there are a few steps to perform in order to run the application.
1) Start an initial "master" node.
Run the following command and get some coffee or watch puppet deploy the master node stack.
$ cd vagrant /vagrant $ vagrant up
2) Start a "deployer" process on the host machine
The deployer process will consume command messages to start and stop virtual machines.
$ app/console rabbitmq:consumer deployer
3) Start a "scaler" policy on the master node
$ cd vagrant /vagrant $ vagrant ssh # now you're in the VM, /data contains the app vagrant@master $ /data/app/console alvi:image-processor:scaler --scalerpolicy queuesize
Now you have to wait until the worker comes up. You can do this by checking out the graphs page (see below) or looking at the VirtualBox gui.
4) Finally start a benchmark
$ cd vagrant /vagrant $ vagrant ssh # now you're in the VM, /data contains the app vagrant@master $ /data/app/console alvi:image-processor:jobSubmit /data/burstWorkload.log
5) Watch some graphs!
graphs.html file in the
doc directory in your browser and watch
the graphs as the system scales up and down.
When you are done with playing with the system, open the VirtualBox gui to remove all created virtual machines that were not already destroyed.
/data/app/console command for other possible commands such as
inspecting the contents of zookeeper.