Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


Storm-yarn enables Storm clusters to be deployed into machines managed by Hadoop YARN. It is still a work in progress.


Mailing list

Feel free to ask questions on storm-yarn's mailing list:

New features:

Based on the project developed by yahoo, we have added following new features.

  1. We have updated the version of Apache Storm from 0.9.0 to 1.0.1.

  2. We have added StormClusterChecker class, in order to monitor the storm cluster. It can adjust the number of supervisors based on the usage of system resources.

  3. We have added the function, namely removeSupervisors() in order to monitor resources. Its functionality is just opposite to that of addSupervisors().

  4. We have updated the logging framework from logback to log4j2.

How to install and use


  • Install Java 8 and Maven 3 first. These two software are necessary to compiling and packaging the source code of storm-on-yarn.

  • Make sure Hadoop YARN have been properly launched.

  • The storm-on-yarn implementation does not include running Zookeeper on YARN. Make sure the Zookeeper service is independently launched beforehands.

Download and build

  1. Download the source code of storm-on-yarn, e.g., execute the command git clone <link> to get the source code.

  2. Edit pom.xml in storm-on-yarn root directory to set the Hadoop version.


  1. To package items, please execute the following command under storm-on-yarn root directory.

     mvn package

You will see the following execution messages.

17:57:27.810 [main] INFO - bin/storm-yarn launch ./conf/storm.yaml --stormZip lib/ --appname storm-on-yarn-test --output target/appId.txt
  17:57:59.681 [main] INFO - bin/storm-yarn getStormConfig ./conf/storm.yaml --appId application_1372121842369_0001 --output ./lib/storm/storm.yaml
  17:58:04.382 [main] INFO - ./lib/storm/bin/storm jar lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.ExclamationTopology exclamation-topology
  17:58:04.382 [main] INFO - ./lib/storm/bin/storm kill exclamation-topology
  17:58:07.798 [main] INFO - bin/storm-yarn stopNimbus ./conf/storm.yaml --appId application_1372121842369_0001
  17:58:10.131 [main] INFO - bin/storm-yarn startNimbus ./conf/storm.yaml --appId application_1372121842369_0001
  17:58:12.460 [main] INFO - bin/storm-yarn stopUI ./conf/storm.yaml --appId application_1372121842369_0001
  17:58:15.045 [main] INFO - bin/storm-yarn startUI ./conf/storm.yaml --appId application_1372121842369_0001
  17:58:17.390 [main] INFO - bin/storm-yarn shutdown ./conf/storm.yaml --appId application_1372121842369_0001

If you want to skip the tests, please add -DskipTests .

   mvn package -DskipTests


After compiling and building the whole project of storm-on-yarn, next you need to install storm-on-yarn and Storm on the Storm Client machine, which is used for submitting the YARN applications to YARN ResourceManager (RM) later.

Please refer to the following  guide, step by step to deploy on the Storm Client machine.

  1. Copy the packaged storm-on-yarn project to Storm Client machine, downloading the project of storm-1.0.1. and put the decompressed project of storm-1.0.1 into same directory as the storm-on-yarn project. As shown below,


So far, you have put storm-on-yarn and storm in the right place on Storm Client machine. You do not need to start running the Storm cluster, as this will be done by running storm-on-yarn later on.

  1. When executing storm-on-yarn commands, commands like "storm-yarn", "storm" and etc., will be frequently called. Therefore, all paths to the bin files containing these executable commands must be included to the PATH environment variable.

Hence you are suggested to add storm-1.0.1/bin and $(storm-on-yarn root directory)/bin to your PATH environment variable, like this:


  1. Storm-on-yarn will replicate a copy of Storm code throughout all the nodes of the YARN cluster using HDFS. However, the location of where to fetch such copy is hard-coded into the Storm-on-YARN client. Therefore, you will have to manually prepare the copy inside HDFS.
     The file (the copy of Storm code) can be stored in HDFS under path "/lib/storm/[storm version]/".

Following commands illustrate how to upload the from the local directory to "/lib/storm/1.0.1" in HDFS.

  hadoop fs -mkdir /lib 
  hadoop fs -mkdir /lib/storm
  hadoop fs -mkdir /lib/storm/1.0.1
  zip -r storm-1.0.1
  hadoop fs -put /lib/storm/1.0.1/

At this point, the deployment step is complete.


Everything should be ready. Now you can start your storm-on-yarn project.

The storm-on-yarn project have a set of specify commands and it use storm-yarn [command] -[arg] xxx as the comman format.

To launch the cluster you can run

storm-yarn launch [storm-yarn-configuration]

[storm-yarn-configuration], which is usually a .yaml file, including all the required configurations during the launch of the Storm cluster.

In this project, we provide two quick ways to create the storm-yarn-configuration file:

a) Edit the storm.yaml file under storm-1.0.1/conf folder

b) Copy the $(storm-on-yarn root directory)/src/main/master_defaults.yaml to storm-1.0.1/conf and rename it to master.yaml, and then edit it where necessary.

In the simplest case, the only configuration you need to add is the Zookeeper cluster information and set the port of supervisor:


When you write a right configure file and execute the command above, the Hadoop YARN will return a application ID, if you deploy completely right. like this:


And, you can see the graphic below on YARN UI and Storm UI respectively.



So far, the storm-on-yarn has run and you can submit topology.

You can run the following command to submit a topology.

  storm jar ***.jar topologyName <arg> -c nimbus=

For example :

  storm jar ~/WordCount2/testWord-1.0-SNAPSHOT.jar storm.topology.WordCountTopology wordcount -c nimbus=

Other details:

  1. storm-yarn has a number of new options for configuring the storm ApplicationManager (AM), e.g.,

    • master.initial-num-supervisors, which stands for the initial number of supervisors to launch with storm.
    • master.container.size-mb, which stands for the size of the containers to request (from the YARN RM).
  2. The procedure of "storm-yarn launch" returns an Application ID, which uniquely identifies the newly launched Storm master. This Application ID will be used for accessing the Storm master.

To obtain a storm.yaml from the newly launch Storm master, you can run

  storm-yarn getStormConfig <storm-yarn-config> --appId <Application-ID> --output <storm.yaml>

storm.yaml will be retrieved from Storm master.

  1. For a full list of storm-yarn commands and options you can run

     storm-yarn help
  2. Storm-on-yarn is now configured to use Netty for communication between spouts and bolts.

It's pure JVM based, and thus OS independent.

If you are running storm using zeromq (instead of Netty), you need to augment the standard file the needed .so files. This can be done with the not ideally named script
  1. Ideally the file is a world readable file installed by ops so there is only one copy in the distributed cache ever.


Command Usage
storm-yarn launch launch storm on yarn
storm-yarn help get help
storm-yarn version view storm version
storm-yarn addSupervisors/removeSupervisor add/remove supervisor
storm-yarn startNimbus/stopNimbus start/stop Nimbus
storm-yarn startUI/stopUI start/stop Web UI
storm-yarn startSupervisors/stopSupervisor start or stop all supervisors
storm-yarn shutdown shutdown storm cluster


Argument Usage
-appname (Only for storm-yarn launch) Application Name. Default value – "Storm-on-Yarn"
-appId (Required) The storm clusters app ID
-output Output file
-container (Required for removeSupervisors) the supervisor to be removed
-supervisors (Required for addSupervisors) The # of supervisors to be added

Known Issues:

There is no failover when nimbus goes down. Still working on it.

There is no simple way to get to the logs for the different processes.


The use and distribution terms for this software are covered by the Apache License, Version 2.0 (


This work was partially supported by SingAREN/AWS Cloud Credit for Research Program 2016.


Storm-yarn enables Storm clusters to be deployed into machines managed by Hadoop YARN.



Code of conduct





No releases published


No packages published