Storm for Yarn
Switch branches/tags
Nothing to show
Clone or download
revans2 Merge pull request #84 from vesense/master
Upgrade Storm to the latest 0.9.x branch(0.9.6)
Latest commit f2fcb2c Jan 25, 2016


Storm-yarn enables Storm clusters to be deployed into machines managed by Hadoop YARN. It is still a work in progress.


Mailing list

Feel free to ask questions on storm-yarn's mailing list:


Please install the following software first:

  • Java 7


To run the tests, you execute the following command.

mvn package

You will see that storm-yarn commands being executed.

17:57:27.810 [main] INFO - bin/storm-yarn launch ./conf/storm.yaml --stormZip lib/ --appname storm-on-yarn-test --output target/appId.txt
17:57:59.681 [main] INFO - bin/storm-yarn getStormConfig ./conf/storm.yaml --appId application_1372121842369_0001 --output ./lib/storm/storm.yaml
17:58:04.382 [main] INFO - ./lib/storm/bin/storm jar lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.ExclamationTopology exclamation-topology
17:58:04.382 [main] INFO - ./lib/storm/bin/storm kill exclamation-topology
17:58:07.798 [main] INFO - bin/storm-yarn stopNimbus ./conf/storm.yaml --appId application_1372121842369_0001
17:58:10.131 [main] INFO - bin/storm-yarn startNimbus ./conf/storm.yaml --appId application_1372121842369_0001
17:58:12.460 [main] INFO - bin/storm-yarn stopUI ./conf/storm.yaml --appId application_1372121842369_0001
17:58:15.045 [main] INFO - bin/storm-yarn startUI ./conf/storm.yaml --appId application_1372121842369_0001
17:58:17.390 [main] INFO - bin/storm-yarn shutdown ./conf/storm.yaml --appId application_1372121842369_0001

If you want to skip the tests you can run

mvn package -DskipTests


You need to install a version of storm on the hadoop gateway.

You also need to place a corresponding file in HDFS so it can be shipped to all of the nodes through the distributed cache at


Storm-YARN is now configured to use Netty for communication between spouts and bolts. It's pure JVM based, and thus OS independent.

If you are running storm using zeromq (instead of Netty), you need to augment the standard file the needed .so files. This can be done with the not ideally named script

Ideally the file is a world readable file installed by ops so there is only one copy in the distributed cache ever.


The yarn-storm command provides a way to launch a storm cluster. In the future it is intended to also provide ways to manage the cluster.

To launch a cluster you can run

storm-yarn launch <storm-yarn-config>

storm-yarn-configuration (a yaml file) will be used to launch a Storm cluster. storm-yarn has a number of new config options to configure the storm AM.

  • master.initial-num-supervisors is the number of supervisors to launch with storm.
  • master.container.size-mb is the size of the container to request. "storm-yarn launch" produces an Application ID, which identify the newly launched Storm master. This Application ID should be used for accessing the Storm master.

To obtain a storm.yaml from the newly launch Storm master, you can run

storm-yarn getStormConfig <storm-yarn-config> --appId <Application-ID> --output <storm.yaml>

storm.yaml will be retrieved from Storm master.

After storing the above storm.yaml in Storm classpath (ex. ~/.storm/storm.yaml), you will invoke standard Storm commands against the Storm cluster on YARN. For example, you run the following command to submit a topology

storm jar <appJar>

For a full list of storm-yarn commands and options you can run

storm-yarn help

Known Issues:

There is no failover when nimbus goes down. Still working on it.

There is no simple way to get to the logs for the different processes.


The use and distribution terms for this software are covered by the Apache License, Version 2.0 (