Home

Table of Contents Installation of the Storm distributed realtime computation system JZMQ Running Storm UI

Installation of the Storm distributed realtime computation system

Currently, we're in the process of installing a Storm network on a local mini cluster at the university. The cluster consists of a handful nodes all running Ubuntu 12.04 with OpenJDK 6. Storm is installed and run by a dedicated user. It requires the following things to work:

• ZooKeeper mini-cluster (only nimbus will run a ZooKeeper instance in our setup!)
• ZeroMQ
• JZMQ (java bindings for ZeroMQ)
• Supervisor for easy recovery of both ZooKeeper and Storm. We use daemontools for simplicity.
• Configuration

I tried to compact the installation procedure into this simple bash-script. The basic steps are:

• Log in to target host.
• Check for sudo access & install git.

 $ sudo apt-get install git

• Clone the script.

 $ git clone git://github.com/maltoe/storm-install.git

• Run the script, executing all phases and setting nimbus.

 $ ./storm-install/storm_install.sh all <nimbus> <installdir>

Alternatively, you could probably run the script via ssh directly.

JZMQ

The JZMQ fork that should be used according to the storm wiki fails to compile with the more recent autoconf and libtools versions shipped with Ubuntu 12.04. Thus, I used the upstream version and in the script calibrated the commit to e2dd65. Seems to work.

Running

• If all went fine (read the log!), execute start.sh to run storm. I prefer to do this using a screen session, but it should be fine to use nohup instead. Of course, if you want to run storm on boot, you can also put start.sh in some init script (e.g., /etc/rc.local).

 $ screen
 $ cd <installdir>
 $ ./start.sh
 [Ctrl-A Ctrl-D]

• Now, the storm node is running in a detached screen session. You can close the ssh connection. To shutdown the node, you can either re-attach to the screen session,

 $ screen -r
 [Ctrl-C]

or you can run the provided stop.sh script to simply shutdown the daemontools supervisor.

 $ <installdir>/stop.sh

Storm UI

For administration of the storm cluster, you can run storm ui on Nimbus, for example:

 [ssh to nimbus]
 $ cd <installdir>/storm/storm-0.8.1
 $ bin/storm ui
 [point your browser to http://<nimbus>:8080/]

For now, I did not provide a script to run the UI in supervisory mode (i.e., using daemontools), although this is stated on the Storm website. However, I could not see why the UI should fail at all, as well as I did not want the UI to run indefinitely. If you want to run the UI continuously, consider using screen or nohup again.