Tools for installing, starting, and destroying HBase, Accumulo, and Storm slider apps on YARN.
WARNING: This is intended for POC/sandbox testing, may not be idempotent so DO NOT use on an existing Production cluster!!!
- Clone the repo
- Set the necessary cluster variables in slider_setup.conf, it shouldn't be necessary to change any other values but thoe ones below
# Zookeeper nodes
ZK_QUORUM="zk1:2181,zk2:2181,zk3:2181"
# Resource Manager address (yarn.resourcemanager.address)
RM_ADDRESS="rm1:8050"
# Resource Manager scheduler address (yarn.resourcemanager.scheduler.address)
RM_SCHED_ADDRESS="rm1:8030"
# Default FS (fs.defaultFS)
DEFAULT_FS="hdfs://nn1:8020"
-
slider_setup is the main script and handles the following
- Pulls down slider and extracts the contents to the SLIDER_INST_DIR
- Modifies slider-client.xml with cluster related info
- Pulls down the slider enabled version of the specified product
- Creates necessary directories and copies required files to HDFS
- For HBase, creates the app dir in HDFS
- Submits the slider base application to the YARN cluster
-
The following args are required
- -f - The path to the slider_setup.conf that has been modified with cluster info
- -p - The product to run (hbase, accumulo, or storm are all that are supported at this time)
- -w - The number of "worker" nodes. This has different meaning depending on product.
- HBase - number of region servers
- Accumulo - number of tablet servers
- Storm - number of supervisors
- -n - The name of the app, this will be the display name in the resource manager and is used by the teardown process
-
HBase Example:
./slider_setup -f slider_setup.conf -p hbase -w 5 -n hbase-slider
- Accumulo Example:
./slider_setup -f slider_setup.conf -p accumulo -w 3 -n accumulo-slider
- Storm Example:
./slider_setup -f slider_setup.conf -p storm -w 3 -n storm-slider
-
slider_destory will do the following
- Freeze the slider application based on provided name
- Destory the slider application based on provided name
-
The following args are required
- -f - The path to the slider_setup.conf that has been modified with cluster info
- -n - The name of the app, this was provided to the slider_setup tool
-
HBase Example:
./slider_destroy -f slider_setup.conf -n hbase-slider
- Accumulo Example:
./slider_destroy -f slider_setup.conf -n accumulo-slider
- Storm Example:
./slider_destroy -f slider_setup.conf -n storm-slider