Browse files

initial commit

  • Loading branch information...
0 parents commit 735190118777697eaf63bca78a2e40f49230b37a @apurtell committed May 13, 2010
104 README.txt
@@ -0,0 +1,104 @@
+HBase EC2
+
+This collection of scripts allows you to run HBase clusters on Amazon.com's Elastic Compute Cloud (EC2) service described at:
+
+ http://aws.amazon.com/ec2
+
+To get help, type the following in a shell:
+
+ bin/hbase-ec2
+
+You need both the EC2 API and AMI tools
+
+ http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351
+
+ http://developer.amazonwebservices.com/connect/entry.jspa?externalID=368&categoryID=88
+
+installed and on the path.
+
+When setting up keypairs on EC2, be sure to name your keypair as 'root'.
+
+Quick Start:
+
+1) Download and unzip the EC2 AMI and API tools zipfiles.
+
+ For Ubuntu, "apt-get install ec2-ami-tools ec2-api-tools".
+
+2) Put the tools on the path and set EC2_HOME in the environment to point to
+ the top level directory of the API tools.
+
+3) Configure src/contrib/ec2/bin/hbase-ec2-env.sh
+
+ Fill in AWS_ACCOUNT_ID with your EC2 account number.
+
+ Fill in AWS_ACCESS_KEY_ID with your EC2 access key.
+
+ Fill in AWS_SECRET_ACCESS_KEY with your EC2 secret access key.
+
+ Fill in EC2_PRIVATE_KEY with the location of your AWS private key file --
+ must begin with 'pk' and end with '.pem'.
+
+ Fill in EC2_CERT with the location of your AWS certificate -- must begin
+ with 'cert' and end with '.pem'.
+
+ Make sure the private part of your AWS SSH keypair exists in the same
+ directory as EC2_PRIVATE_KEY with the name id_rsa_root. Also, insure that
+ the permissions on the private key file are 600 (ONLY owner readable/
+ writable).
+
+4) ./bin/hbase-ec2 launch-cluster <name> <nr-zoos> <nr-slaves>, e.g
+
+ ./bin/hbase-ec2 launch-cluster testcluster 3 3
+
+5) Once the above command has finished without error, ./bin/hbase-ec2 login
+ <name>, e.g.
+
+ ./bin/hbase-ec2 login testcluster
+
+6) Check that the cluster is up and functional:
+
+ hbase shell
+ > status 'simple'
+
+ You should see something like:
+
+ 3 live servers
+ domU-12-31-39-09-75-11.compute-1.internal:60020 1258653694915
+ requests=0, regions=1, usedHeap=29, maxHeap=987
+ domU-12-31-39-01-AC-31.compute-1.internal:60020 1258653709041
+ requests=0, regions=1, usedHeap=29, maxHeap=987
+ domU-12-31-39-01-B0-91.compute-1.internal:60020 1258653706411
+ requests=0, regions=0, usedHeap=27, maxHeap=987
+ 0 dead servers
+
+
+Extra Packages:
+
+It is possible to specify that extra packages be downloaded and installed on
+demand when the master and slave instances start.
+
+ 1. Set up a YUM repository. See: http://yum.baseurl.org/wiki/RepoCreate
+
+ 2. Host the repository somewhere public. For example, build the
+ repository locally and then copy it up to an S3 bucket.
+
+ 3. Create a YUM repository descriptor (.repo file). See:
+ http://yum.baseurl.org/wiki/RepoCreate
+
+ [myrepo]
+ name = MyRepo
+ baseurl = http://mybucket.s3.amazonaws.com/myrepo
+ enabled = 1
+
+ Upload the .repo file somewhere public, for example, in the root
+ directory of the repository,
+ mybucket.s3.amazonaws.com/myrepo/myrepo.repo
+
+ 4. Configure hbase-ec2-env.sh thus:
+
+ EXTRA_PACKAGES="http://mybucket.s3.amazonaws.com/myrepo/myrepo.repo \
+ pkg1 pkg2 pkg3"
+
+When the master and slave instances start, the .repo file will be added to
+the Yum repository list and then Yum will be invoked to pull the packages
+listed after the URL.
69 bin/cmd-hbase-cluster
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Run commands on master or specified node of a running HBase EC2 cluster.
+
+# if no args specified, show usage
+if [ $# = 0 ]; then
+ echo "Command required!"
+ exit 1
+fi
+
+# get arguments
+COMMAND="$1"
+shift
+# get group
+CLUSTER="$1"
+shift
+
+if [ -z $CLUSTER ]; then
+ echo "Cluster name or instance id required!"
+ exit 1
+fi
+
+# Import variables
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+. "$bin"/hbase-ec2-env.sh
+
+if [[ "$CLUSTER" = "i-*" ]]; then
+ HOST=`ec2-describe-instances $TOOL_OPTS $CLUSTER | grep running | awk '{print $4}'`
+ [ -z $HOST ] && echo "Instance still pending or no longer running: $CLUSTER" && exit 1
+else
+ [ ! -f $MASTER_IP_PATH ] && echo "Wrong group name, or cluster not launched! $CLUSTER" && exit 1
+ HOST=`cat $MASTER_IP_PATH`
+fi
+
+if [ "$COMMAND" = "login" ] ; then
+ echo "Logging in to host $HOST."
+ ssh $SSH_OPTS "root@$HOST"
+elif [ "$COMMAND" = "proxy" ] ; then
+ echo "Proxying to host $HOST via local port 6666"
+ echo "Gangia: http://$HOST/ganglia"
+ echo "JobTracker: http://$HOST:50030/"
+ echo "NameNode: http://$HOST:50070/"
+ ssh $SSH_OPTS -D 6666 -N "root@$HOST"
+elif [ "$COMMAND" = "push" ] ; then
+ echo "Pushing $1 to host $HOST."
+ scp $SSH_OPTS -r $1 "root@$HOST:"
+elif [ "$COMMAND" = "screen" ] ; then
+ echo "Logging in and attaching screen on host $HOST."
+ ssh $SSH_OPTS -t "root@$HOST" 'screen -D -R'
+else
+ echo "Executing command on host $HOST."
+ ssh $SSH_OPTS -t "root@$HOST" "$COMMAND"
+fi
84 bin/create-hbase-image
@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Create a HBase AMI.
+# Inspired by Jonathan Siegel's EC2 script (http://blogsiegel.blogspot.com/2006/08/sandboxing-amazon-ec2.html)
+
+# allow override of SLAVE_INSTANCE_TYPE from the command line
+[ ! -z $1 ] && SLAVE_INSTANCE_TYPE=$1
+
+# Import variables
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+. "$bin"/hbase-ec2-env.sh
+
+type=$SLAVE_INSTANCE_TYPE
+arch=$SLAVE_ARCH
+
+echo "INSTANCE_TYPE is $type"
+echo "ARCH is $arch"
+
+AMI_IMAGE=`ec2-describe-images $TOOL_OPTS -a | grep $S3_BUCKET | grep hbase | grep $HBASE_VERSION-$arch | grep available | awk '{print $2}'`
+
+[ ! -z $AMI_IMAGE ] && echo "AMI already registered, use: ec2-deregister $AMI_IMAGE" && exit 1
+
+echo "Starting a AMI with ID $BASE_AMI_IMAGE."
+OUTPUT=`ec2-run-instances $BASE_AMI_IMAGE $TOOL_OPTS -k root -t $type`
+BOOTING_INSTANCE=`echo $OUTPUT | awk '{print $6}'`
+
+echo "Instance is $BOOTING_INSTANCE."
+
+echo "Polling server status"
+while true; do
+ printf "."
+ HOSTNAME=`ec2-describe-instances $TOOL_OPTS $BOOTING_INSTANCE | grep running | awk '{print $4}'`
+ if [ ! -z $HOSTNAME ]; then
+ break;
+ fi
+ sleep 1
+done
+echo "The server is available at $HOSTNAME."
+while true; do
+ REPLY=`ssh $SSH_OPTS "root@$HOSTNAME" 'echo "hello"'`
+ if [ ! -z $REPLY ]; then
+ break;
+ fi
+ sleep 5
+done
+
+echo "Copying scripts."
+
+# Copy setup scripts
+scp $SSH_OPTS "$bin"/hbase-ec2-env.sh "root@$HOSTNAME:/mnt"
+scp $SSH_OPTS "$bin"/functions.sh "root@$HOSTNAME:/mnt"
+if [ -f "$bin"/credentials.sh ] ; then
+ scp $SSH_OPTS "$bin"/credentials.sh "root@$HOSTNAME:/mnt"
+fi
+scp $SSH_OPTS "$bin"/image/create-hbase-image-remote "root@$HOSTNAME:/mnt"
+scp $SSH_OPTS "$bin"/image/ec2-run-user-data "root@$HOSTNAME:/etc/init.d"
+
+# Copy private key and certificate (for bundling image)
+scp $SSH_OPTS $EC2_PRIVATE_KEY "root@$HOSTNAME:/mnt"
+scp $SSH_OPTS $EC2_CERT "root@$HOSTNAME:/mnt"
+
+# Connect to it
+ssh $SSH_OPTS "root@$HOSTNAME" "sh -c \"INSTANCE_TYPE=$type ARCH=$arch /mnt/create-hbase-image-remote\""
+
+# Register image
+ec2-register $TOOL_OPTS -n hbase-$REGION-$HBASE_VERSION-$arch $S3_BUCKET/hbase-$HBASE_VERSION-$arch.manifest.xml
+
+echo "Terminate with: ec2-terminate-instances $BOOTING_INSTANCE"
47 bin/delete-hbase-cluster
@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Delete the groups an local files associated with a cluster.
+
+if [ -z $1 ]; then
+ echo "Cluster name required!"
+ exit 1
+fi
+
+CLUSTER=$1
+
+# Import variables
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+. "$bin"/hbase-ec2-env.sh
+
+# Finding HBase clusters
+CLUSTERS=`ec2-describe-instances $TOOL_OPTS | \
+ awk '"RESERVATION" == $1 && $4 ~ /-master$/, "INSTANCE" == $1' | tr '\n' '\t' | \
+ grep "$CLUSTER" | grep running | cut -f4 | rev | cut -d'-' -f2- | rev`
+
+if [ -n "$CLUSTERS" ]; then
+ echo "Cluster $CLUSTER has running instances. Please terminate them first."
+ exit 0
+fi
+
+"$bin"/revoke-hbase-cluster-secgroups $CLUSTER
+
+rm -f $MASTER_IP_PATH
+rm -f $MASTER_PRIVATE_IP_PATH
+rm -f $MASTER_ZONE_PATH
+rm -f $ZOOKEEPER_QUORUM_PATH
39 bin/functions.sh
@@ -0,0 +1,39 @@
+#
+# Copyright 2010 The Apache Software Foundation
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+function getCredentialSetting {
+ name=$1
+ eval "val=\$$name"
+ if [ -z "$val" ] ; then
+ if [ -f "$bin"/credentials.sh ] ; then
+ val=`cat "$bin"/credentials.sh | grep $name | awk 'BEGIN { FS="=" } { print $2; }'`
+ if [ -z "$val" ] ; then
+ echo -n "$name: "
+ read -e val
+ echo "$name=$val" >> "$bin"/credentials.sh
+ fi
+ else
+ echo -n "$name: "
+ read -e val
+ echo "$name=$val" >> "$bin"/credentials.sh
+ fi
+ eval "$name=$val"
+ fi
+}
63 bin/hbase-ec2
@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+# if no args specified, show usage
+if [ $# = 0 ]; then
+ echo "Usage: hbase-ec2 COMMAND"
+ echo "where COMMAND is one of:"
+ echo " list list all running HBase EC2 clusters"
+ echo " launch-cluster <name> <slaves> <zoos> launch a HBase cluster"
+ echo " launch-zookeeper <name> <zoos> launch the zookeeper quorum"
+ echo " launch-master <name> launch or find a cluster master"
+ echo " launch-slaves <name> <slaves> launch the cluster slaves"
+ echo " terminate-cluster <name> terminate all HBase EC2 instances"
+ echo " delete-cluster <name> clean up after a terminated cluster"
+ echo " login <name|instance id> login to the master node"
+ echo " screen <name|instance id> start or attach 'screen' on the master"
+ echo " proxy <name|instance id> start a socks proxy on localhost:6666"
+ echo " push <name> <file> scp a file to the master node"
+ echo " <shell cmd> <group|instance id> execute a command on the master"
+ echo " create-image create a HBase AMI"
+ exit 1
+fi
+
+# get arguments
+COMMAND="$1"
+shift
+
+if [ "$COMMAND" = "create-image" ] ; then
+ . "$bin"/create-hbase-image $*
+elif [ "$COMMAND" = "launch-cluster" ] ; then
+ . "$bin"/launch-hbase-cluster $*
+elif [ "$COMMAND" = "launch-zookeeper" ] ; then
+ . "$bin"/launch-hbase-zookeeper $*
+elif [ "$COMMAND" = "launch-master" ] ; then
+ . "$bin"/launch-hbase-master $*
+elif [ "$COMMAND" = "launch-slaves" ] ; then
+ . "$bin"/launch-hbase-slaves $*
+elif [ "$COMMAND" = "delete-cluster" ] ; then
+ . "$bin"/delete-hbase-cluster $*
+elif [ "$COMMAND" = "terminate-cluster" ] ; then
+ . "$bin"/terminate-hbase-cluster $*
+elif [ "$COMMAND" = "list" ] ; then
+ . "$bin"/list-hbase-clusters
+else
+ . "$bin"/cmd-hbase-cluster "$COMMAND" $*
+fi
133 bin/hbase-ec2-env.sh
@@ -0,0 +1,133 @@
+# Set environment variables for running Hbase on Amazon EC2 here.
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Your Amazon Account Number.
+AWS_ACCOUNT_ID=
+
+# Your Amazon AWS access key.
+AWS_ACCESS_KEY_ID=
+
+# Your Amazon AWS secret access key.
+AWS_SECRET_ACCESS_KEY=
+
+# Your AWS private key file -- must begin with 'pk' and end with '.pem'
+EC2_PRIVATE_KEY=
+
+# Your AWS certificate file -- must begin with 'cert' and end with '.pem'
+EC2_CERT=
+
+# Your AWS root SSH key
+EC2_ROOT_SSH_KEY=
+
+# The version of HBase to use.
+HBASE_VERSION=0.20.4
+
+# The version of Hadoop to use.
+HADOOP_VERSION=0.20.2
+
+# The Amazon EC2 bucket for images
+REGION=us-east-1
+#REGION=us-west-1
+#REGION=eu-west-1
+#REGION=ap-southeast-1
+S3_BUCKET=apache-hbase-images-$REGION
+# Account for bucket
+# We need this because S3 is returning account identifiers instead of bucket
+# names.
+S3_ACCOUNT=720040977164
+
+# Enable public access web interfaces
+ENABLE_WEB_PORTS=false
+
+# Extra packages
+# Allows you to add a private Yum repo and pull packages from it as your
+# instances boot up. Format is <repo-descriptor-URL> <pkg1> ... <pkgN>
+# The repository descriptor will be fetched into /etc/yum/repos.d.
+EXTRA_PACKAGES=
+
+# Use only c1.xlarge unless you know what you are doing
+MASTER_INSTANCE_TYPE=${MASTER_INSTANCE_TYPE:-c1.xlarge}
+
+# Use only c1.xlarge unless you know what you are doing
+SLAVE_INSTANCE_TYPE=${SLAVE_INSTANCE_TYPE:-c1.xlarge}
+
+# Use only c1.medium unless you know what you are doing
+ZOO_INSTANCE_TYPE=${ZOO_INSTANCE_TYPE:-c1.medium}
+
+############################################################################
+
+# Generally, users do not need to edit below
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+. "$bin"/functions.sh
+getCredentialSetting 'AWS_ACCOUNT_ID'
+getCredentialSetting 'AWS_ACCESS_KEY_ID'
+getCredentialSetting 'AWS_SECRET_ACCESS_KEY'
+getCredentialSetting 'EC2_PRIVATE_KEY'
+getCredentialSetting 'EC2_CERT'
+getCredentialSetting 'EC2_ROOT_SSH_KEY'
+
+export EC2_URL=https://$REGION.ec2.amazonaws.com
+
+# SSH options used when connecting to EC2 instances.
+SSH_OPTS=`echo -q -i "$EC2_ROOT_SSH_KEY" -o StrictHostKeyChecking=no -o ServerAliveInterval=30`
+
+# EC2 command request timeout (seconds)
+REQUEST_TIMEOUT=300 # 5 minutes
+
+# Global tool options
+TOOL_OPTS=`echo -K "$EC2_PRIVATE_KEY" -C "$EC2_CERT" --request-timeout $REQUEST_TIMEOUT`
+
+# The EC2 group master name. CLUSTER is set by calling scripts
+CLUSTER_MASTER=$CLUSTER-master
+
+# Cached values for a given cluster
+MASTER_PRIVATE_IP_PATH=~/.hbase-private-$CLUSTER_MASTER
+MASTER_IP_PATH=~/.hbase-$CLUSTER_MASTER
+MASTER_ZONE_PATH=~/.hbase-zone-$CLUSTER_MASTER
+
+# The Zookeeper EC2 group name. CLUSTER is set by calling scripts.
+CLUSTER_ZOOKEEPER=$CLUSTER-zookeeper
+ZOOKEEPER_QUORUM_PATH=~/.hbase-quorum-$CLUSTER_ZOOKEEPER
+
+# The script to run on instance boot.
+USER_DATA_FILE=hbase-ec2-init-remote.sh
+
+# The version number of the installed JDK.
+JAVA_VERSION=1.6.0_20
+
+# SUPPORTED_ARCHITECTURES = ['i386', 'x86_64']
+if [ "$SLAVE_INSTANCE_TYPE" = "m1.small" -o "$SLAVE_INSTANCE_TYPE" = "c1.medium" ]; then
+ SLAVE_ARCH='i386'
+ BASE_AMI_IMAGE=${BASE_AMI_IMAGE:-"ami-48aa4921"} # ec2-public-images/fedora-8-i386-base-v1.10.manifest.xml
+else
+ SLAVE_ARCH='x86_64'
+ BASE_AMI_IMAGE=${BASE_AMI_IMAGE:-"ami-f61dfd9f"} # ec2-public-images/fedora-8-x86_64-base-v1.10.manifest.xml
+fi
+
+if [ "$MASTER_INSTANCE_TYPE" = "m1.small" -o "$MASTER_INSTANCE_TYPE" = "c1.medium" ]; then
+ MASTER_ARCH='i386'
+else
+ MASTER_ARCH='x86_64'
+fi
+
+if [ "$ZOO_INSTANCE_TYPE" = "m1.small" -o "$ZOO_INSTANCE_TYPE" = "c1.medium" ]; then
+ ZOO_ARCH='i386'
+else
+ ZOO_ARCH='x86_64'
+fi
264 bin/hbase-ec2-init-remote.sh
@@ -0,0 +1,264 @@
+#!/usr/bin/env bash
+
+#
+# Copyright 2010 The Apache Software Foundation
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Script that is run on each EC2 instance on boot. It is passed in the EC2 user
+# data, so should not exceed 16K in size.
+
+MASTER_HOST="%MASTER_HOST%"
+ZOOKEEPER_QUORUM="%ZOOKEEPER_QUORUM%"
+EXTRA_PACKAGES="%EXTRA_PACKAGES%"
+SECURITY_GROUPS=`wget -q -O - http://169.254.169.254/latest/meta-data/security-groups`
+IS_MASTER=`echo $SECURITY_GROUPS | awk '{ a = match ($0, "-master$"); if (a) print "true"; else print "false"; }'`
+if [ "$IS_MASTER" = "true" ]; then
+ MASTER_HOST=`wget -q -O - http://169.254.169.254/latest/meta-data/local-hostname`
+fi
+HADOOP_HOME=`ls -d /usr/local/hadoop-*`
+HADOOP_VERSION=`echo $HADOOP_HOME | cut -d '-' -f 2`
+HBASE_HOME=`ls -d /usr/local/hbase-*`
+HBASE_VERSION=`echo $HBASE_HOME | cut -d '-' -f 2`
+
+export USER="root"
+
+# up file-max
+sysctl -w fs.file-max=65535
+
+# up ulimits
+echo "root soft nofile 65535" >> /etc/security/limits.conf
+echo "root hard nofile 65535" >> /etc/security/limits.conf
+ulimit -n 65535
+
+# up epoll limits; ok if this fails, only valid for kernels 2.6.27+
+sysctl -w fs.epoll.max_user_instances=65535 > /dev/null 2>&1
+
+[ ! -f /etc/hosts ] && echo "127.0.0.1 localhost" > /etc/hosts
+
+# Extra packages
+
+if [ "$EXTRA_PACKAGES" != "" ] ; then
+ # format should be <repo-descriptor-URL> <package1> ... <packageN>
+ pkg=( $EXTRA_PACKAGES )
+ wget -nv -O /etc/yum.repos.d/user.repo ${pkg[0]}
+ yum -y update yum
+ yum -y install ${pkg[@]:1}
+fi
+
+# Ganglia
+
+if [ "$IS_MASTER" = "true" ]; then
+ sed -i -e "s|\( *mcast_join *=.*\)|#\1|" \
+ -e "s|\( *bind *=.*\)|#\1|" \
+ -e "s|\( *mute *=.*\)| mute = yes|" \
+ -e "s|\( *location *=.*\)| location = \"master-node\"|" \
+ /etc/gmond.conf
+ mkdir -p /mnt/ganglia/rrds
+ chown -R ganglia:ganglia /mnt/ganglia/rrds
+ rm -rf /var/lib/ganglia; cd /var/lib; ln -s /mnt/ganglia ganglia; cd
+ service gmond start
+ service gmetad start
+ apachectl start
+else
+ sed -i -e "s|\( *mcast_join *=.*\)|#\1|" \
+ -e "s|\( *bind *=.*\)|#\1|" \
+ -e "s|\(udp_send_channel {\)|\1\n host=$MASTER_HOST|" \
+ /etc/gmond.conf
+ service gmond start
+fi
+
+# Reformat sdb as xfs
+umount /mnt
+mkfs.xfs -f /dev/sdb
+mount -o noatime /dev/sdb /mnt
+
+# Probe for additional instance volumes
+
+# /dev/sdb as /mnt is always set up by base image
+DFS_NAME_DIR="/mnt/hadoop/dfs/name"
+DFS_DATA_DIR="/mnt/hadoop/dfs/data"
+i=2
+for d in c d e f g h i j k l m n o p q r s t u v w x y z; do
+ m="/mnt${i}"
+ mkdir -p $m
+ mkfs.xfs -f /dev/sd${d}
+ if [ $? -eq 0 ] ; then
+ mount -o noatime /dev/sd${d} $m > /dev/null 2>&1
+ if [ $i -lt 3 ] ; then # no more than two namedirs
+ DFS_NAME_DIR="${DFS_NAME_DIR},${m}/hadoop/dfs/name"
+ fi
+ DFS_DATA_DIR="${DFS_DATA_DIR},${m}/hadoop/dfs/data"
+ i=$(( i + 1 ))
+ fi
+done
+
+# Hadoop configuration
+
+cat > $HADOOP_HOME/conf/core-site.xml <<EOF
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+<property>
+ <name>hadoop.tmp.dir</name>
+ <value>/mnt/hadoop</value>
+</property>
+<property>
+ <name>fs.default.name</name>
+ <value>hdfs://$MASTER_HOST:8020</value>
+</property>
+</configuration>
+EOF
+cat > $HADOOP_HOME/conf/hdfs-site.xml <<EOF
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+<property>
+ <name>fs.default.name</name>
+ <value>hdfs://$MASTER_HOST:8020</value>
+</property>
+<property>
+ <name>dfs.name.dir</name>
+ <value>$DFS_NAME_DIR</value>
+</property>
+<property>
+ <name>dfs.data.dir</name>
+ <value>$DFS_DATA_DIR</value>
+</property>
+<property>
+ <name>dfs.datanode.handler.count</name>
+ <value>10</value>
+</property>
+<property>
+ <name>dfs.datanode.max.xcievers</name>
+ <value>10000</value>
+</property>
+</configuration>
+EOF
+cat > $HADOOP_HOME/conf/mapred-site.xml <<EOF
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+<property>
+ <name>mapred.job.tracker</name>
+ <value>$MASTER_HOST:8021</value>
+</property>
+<property>
+ <name>io.compression.codecs</name>
+ <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>
+</property>
+</configuration>
+EOF
+# Update classpath to include HBase jars and config
+cat >> $HADOOP_HOME/conf/hadoop-env.sh <<EOF
+HADOOP_CLASSPATH="$HBASE_HOME/hbase-${HBASE_VERSION}.jar:$HBASE_HOME/lib/zookeeper-3.3.0.jar:$HBASE_HOME/conf"
+EOF
+# Configure Hadoop for Ganglia
+cat > $HADOOP_HOME/conf/hadoop-metrics.properties <<EOF
+dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
+dfs.period=10
+dfs.servers=$MASTER_HOST:8649
+jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
+jvm.period=10
+jvm.servers=$MASTER_HOST:8649
+mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
+mapred.period=10
+mapred.servers=$MASTER_HOST:8649
+EOF
+
+# HBase configuration
+
+cat > $HBASE_HOME/conf/hbase-site.xml <<EOF
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+<property>
+ <name>hbase.rootdir</name>
+ <value>hdfs://$MASTER_HOST:8020/hbase</value>
+</property>
+<property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+</property>
+<property>
+ <name>hbase.zookeeper.quorum</name>
+ <value>$ZOOKEEPER_QUORUM</value>
+</property>
+<property>
+ <name>hbase.regionserver.handler.count</name>
+ <value>100</value>
+</property>
+<property>
+ <name>hbase.hregion.memstore.block.multiplier</name>
+ <value>3</value>
+</property>
+<property>
+ <name>hbase.hstore.blockingStoreFiles</name>
+ <value>15</value>
+</property>
+<property>
+ <name>dfs.replication</name>
+ <value>3</value>
+</property>
+<property>
+ <name>dfs.client.block.write.retries</name>
+ <value>20</value>
+</property>
+<property>
+ <name>zookeeper.session.timeout</name>
+ <value>60000</value>
+</property>
+<property>
+ <name>hbase.tmp.dir</name>
+ <value>/mnt/hbase</value>
+</property>
+</configuration>
+EOF
+# Override JVM options
+cat >> $HBASE_HOME/conf/hbase-env.sh <<EOF
+export HBASE_MASTER_OPTS="-Xmx1000m -XX:+UseConcMarkSweepGC -XX:NewSize=128m -XX:MaxNewSize=128m -XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/mnt/hbase/logs/hbase-master-gc.log"
+export HBASE_REGIONSERVER_OPTS="-Xmx2000m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=88 -XX:NewSize=128m -XX:MaxNewSize=128m -XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/mnt/hbase/logs/hbase-regionserver-gc.log"
+EOF
+# Configure HBase for Ganglia
+cat > $HBASE_HOME/conf/hadoop-metrics.properties <<EOF
+dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
+dfs.period=10
+dfs.servers=$MASTER_HOST:8649
+hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext
+hbase.period=10
+hbase.servers=$MASTER_HOST:8649
+jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
+jvm.period=10
+jvm.servers=$MASTER_HOST:8649
+EOF
+
+mkdir -p /mnt/hadoop/logs /mnt/hbase/logs
+
+if [ "$IS_MASTER" = "true" ]; then
+ # only format on first boot
+ [ ! -e /mnt/hadoop/dfs/name ] && "$HADOOP_HOME"/bin/hadoop namenode -format
+ "$HADOOP_HOME"/bin/hadoop-daemon.sh start namenode
+ "$HADOOP_HOME"/bin/hadoop-daemon.sh start jobtracker
+ "$HBASE_HOME"/bin/hbase-daemon.sh start master
+else
+ "$HADOOP_HOME"/bin/hadoop-daemon.sh start datanode
+ "$HBASE_HOME"/bin/hbase-daemon.sh start regionserver
+ "$HADOOP_HOME"/bin/hadoop-daemon.sh start tasktracker
+fi
+
+rm -f /var/ec2/ec2-run-user-data.*
64 bin/hbase-ec2-init-zookeeper-remote.sh
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+#
+# Copyright 2010 The Apache Software Foundation
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+HBASE_HOME=`ls -d /usr/local/hbase-*`
+
+# HBase configuration (Zookeeper)
+cat > $HBASE_HOME/conf/hbase-site.xml <<EOF
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+<property>
+ <name>hbase.zookeeper.quorum</name>
+ <value>$ZOOKEEPER_QUORUM</value>
+</property>
+<property>
+ <name>zookeeper.session.timeout</name>
+ <value>60000</value>
+</property>
+<property>
+ <name>hbase.zookeeper.property.dataDir</name>
+ <value>/mnt/hbase/zk</value>
+</property>
+<property>
+ <name>hbase.zookeeper.property.maxClientCnxns</name>
+ <value>100</value>
+</property>
+</configuration>
+EOF
+
+# Start services
+
+# up open file descriptor limits
+echo "root soft nofile 32768" >> /etc/security/limits.conf
+echo "root hard nofile 32768" >> /etc/security/limits.conf
+
+# up epoll limits
+# ok if this fails, only valid for kernels 2.6.27+
+sysctl -w fs.epoll.max_user_instance=32768 > /dev/null 2>&1
+
+mkdir -p /mnt/hbase/logs
+mkdir -p /mnt/hbase/zk
+
+[ ! -f /etc/hosts ] && echo "127.0.0.1 localhost" > /etc/hosts
+
+"$HBASE_HOME"/bin/hbase-daemon.sh start zookeeper
114 bin/image/create-hbase-image-remote
@@ -0,0 +1,114 @@
+#!/bin/sh
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Create a Hbase AMI. Runs on the EC2 instance.
+
+# Import variables
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+. "$bin"/hbase-ec2-env.sh
+
+type=$INSTANCE_TYPE
+[ -z "$type" ] && type=$SLAVE_INSTANCE_TYPE
+arch=$ARCH
+[ -z "$arch" ] && arch=$SLAVE_ARCH
+
+echo "Remote: INSTANCE_TYPE is $type"
+echo "Remote: ARCH is $arch"
+
+# Remove sensitive information
+rm -f "$bin"/hbase-ec2-env.sh
+rm -f "$bin"/credentials.sh
+
+# Install Java
+echo "Downloading and installing java binary."
+cd /usr/local
+wget -nv -O java.bin http://hbase.s3.amazonaws.com/jdk/jdk-${JAVA_VERSION}-linux-${arch}.bin
+sh java.bin
+rm -f java.bin
+
+# Install tools
+echo "Installing rpms."
+yum -y update
+yum -y install rsync lynx screen ganglia-gmetad ganglia-gmond ganglia-web httpd php lzo-devel xfsprogs
+yum -y clean all
+
+# Install Hadoop
+echo "Installing Hadoop $HADOOP_VERSION."
+cd /usr/local
+wget -nv http://archive.apache.org/dist/hadoop/core/hadoop-$HADOOP_VERSION/hadoop-$HADOOP_VERSION.tar.gz
+[ ! -f hadoop-$HADOOP_VERSION.tar.gz ] && wget -nv http://www.apache.org/dist/hadoop/core/hadoop-$HADOOP_VERSION/hadoop-$HADOOP_VERSION.tar.gz
+tar xzf hadoop-$HADOOP_VERSION.tar.gz
+rm -f hadoop-$HADOOP_VERSION.tar.gz
+
+# Configure Hadoop
+sed -i \
+ -e "s|# export JAVA_HOME=.*|export JAVA_HOME=/usr/local/jdk${JAVA_VERSION}|" \
+ -e 's|# export HADOOP_LOG_DIR=.*|export HADOOP_LOG_DIR=/mnt/hadoop/logs|' \
+ -e 's|# export HADOOP_SLAVE_SLEEP=.*|export HADOOP_SLAVE_SLEEP=1|' \
+ -e 's|# export HADOOP_OPTS=.*|export HADOOP_OPTS=-server|' \
+ /usr/local/hadoop-$HADOOP_VERSION/conf/hadoop-env.sh
+
+# Install HBase
+echo "Installing HBase $HBASE_VERSION."
+cd /usr/local
+wget -nv http://hbase.s3.amazonaws.com/hbase/hbase-$HBASE_VERSION.tar.gz
+tar xzf hbase-$HBASE_VERSION.tar.gz
+rm -f hbase-$HBASE_VERSION.tar.gz
+
+# Configure HBase
+sed -i \
+ -e "s|# export JAVA_HOME=.*|export JAVA_HOME=/usr/local/jdk${JAVA_VERSION}|" \
+ -e 's|# export HBASE_OPTS=.*|export HBASE_OPTS="$HBASE_OPTS -server -XX:+HeapDumpOnOutOfMemoryError"|' \
+ -e 's|# export HBASE_LOG_DIR=.*|export HBASE_LOG_DIR=/mnt/hbase/logs|' \
+ -e 's|# export HBASE_SLAVE_SLEEP=.*|export HBASE_SLAVE_SLEEP=1|' \
+ /usr/local/hbase-$HBASE_VERSION/conf/hbase-env.sh
+
+# Run user data as script on instance startup
+chmod +x /etc/init.d/ec2-run-user-data
+echo "/etc/init.d/ec2-run-user-data" >> /etc/rc.d/rc.local
+
+# Setup root user bash environment
+
+ echo "export JAVA_HOME=/usr/local/jdk${JAVA_VERSION}" >> /root/.bash_profile
+echo "export HADOOP_HOME=/usr/local/hadoop-${HADOOP_VERSION}" >> /root/.bash_profile
+echo "export HBASE_HOME=/usr/local/hbase-${HBASE_VERSION}" >> /root/.bash_profile
+echo 'export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HBASE_HOME/bin:$PATH' >> /root/.bash_profile
+
+# Configure networking.
+# Delete SSH authorized_keys since it includes the key it was launched with. (Note that it is re-populated when an instance starts.)
+rm -f /root/.ssh/authorized_keys
+# Ensure logging in to new hosts is seamless.
+echo ' StrictHostKeyChecking no' >> /etc/ssh/ssh_config
+
+# Install LZO
+echo "Installing LZO codec support"
+wget -nv -O /tmp/lzo-linux-${HADOOP_VERSION}.tar.gz http://hbase.s3.amazonaws.com/hadoop/lzo-linux-${HADOOP_VERSION}.tar.gz
+cd /usr/local/hadoop-${HADOOP_VERSION} && tar xzf /tmp/lzo-linux-${HADOOP_VERSION}.tar.gz
+cd /usr/local/hbase-${HBASE_VERSION} && tar xzf /tmp/lzo-linux-${HADOOP_VERSION}.tar.gz
+rm -f /tmp/lzo-linux-${HADOOP_VERSION}.tar.gz
+
+# Bundle and upload image
+cd ~root
+# Don't need to delete .bash_history since it isn't written until exit.
+df -h
+ec2-bundle-vol -d /mnt -k /mnt/pk*.pem -c /mnt/cert*.pem -u $AWS_ACCOUNT_ID -s 3072 -p hbase-$HBASE_VERSION-$arch -r $arch
+
+ec2-upload-bundle -b $S3_BUCKET -m /mnt/hbase-$HBASE_VERSION-$arch.manifest.xml -a $AWS_ACCESS_KEY_ID -s $AWS_SECRET_ACCESS_KEY
+
+# End
+echo Done
51 bin/image/ec2-run-user-data
@@ -0,0 +1,51 @@
+#!/bin/bash
+#
+# ec2-run-user-data - Run instance user-data if it looks like a script.
+#
+# Only retrieves and runs the user-data script once per instance. If
+# you want the user-data script to run again (e.g., on the next boot)
+# then add this command in the user-data script:
+# rm -f /var/ec2/ec2-run-user-data.*
+#
+# History:
+# 2008-05-16 Eric Hammond <ehammond@thinksome.com>
+# - Initial version including code from Kim Scheibel, Jorge Oliveira
+# 2008-08-06 Tom White
+# - Updated to use mktemp on fedora
+#
+
+prog=$(basename $0)
+logger="logger -t $prog"
+curl="curl --retry 3 --silent --show-error --fail"
+instance_data_url=http://169.254.169.254/2008-02-01
+
+# Wait until networking is up on the EC2 instance.
+perl -MIO::Socket::INET -e '
+ until(new IO::Socket::INET("169.254.169.254:80")){print"Waiting for network...\n";sleep 1}
+' | $logger
+
+# Exit if we have already run on this instance (e.g., previous boot).
+ami_id=$($curl $instance_data_url/meta-data/ami-id)
+been_run_file=/var/ec2/$prog.$ami_id
+mkdir -p $(dirname $been_run_file)
+if [ -f $been_run_file ]; then
+ $logger < $been_run_file
+ exit
+fi
+
+# Retrieve the instance user-data and run it if it looks like a script
+user_data_file=`mktemp -t ec2-user-data.XXXXXXXXXX`
+chmod 700 $user_data_file
+$logger "Retrieving user-data"
+$curl -o $user_data_file $instance_data_url/user-data 2>&1 | $logger
+if [ ! -s $user_data_file ]; then
+ $logger "No user-data available"
+elif head -1 $user_data_file | egrep -v '^#!'; then
+ $logger "Skipping user-data as it does not begin with #!"
+else
+ $logger "Running user-data"
+ echo "user-data has already been run on this instance" > $been_run_file
+ $user_data_file 2>&1 | logger -t "user-data"
+ $logger "user-data exit code: $?"
+fi
+rm -f $user_data_file
87 bin/init-hbase-cluster-secgroups
@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Set up security groups for the EC2 HBase cluster
+
+if [ -z $1 ]; then
+ echo "Cluster name required!"
+ exit 1
+fi
+
+CLUSTER=$1
+
+# Import variables
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+. "$bin"/hbase-ec2-env.sh
+
+echo "Creating/checking security groups"
+
+ec2-describe-group $TOOL_OPTS | egrep "[[:space:]]$CLUSTER_MASTER[[:space:]]" > /dev/null
+if [ ! $? -eq 0 ]; then
+ echo "Creating group $CLUSTER_MASTER"
+ ec2-add-group $TOOL_OPTS $CLUSTER_MASTER -d "Group for HBase Master."
+ ec2-authorize $TOOL_OPTS $CLUSTER_MASTER -o $CLUSTER_MASTER -u $AWS_ACCOUNT_ID
+ ec2-authorize $TOOL_OPTS $CLUSTER_MASTER -p 22 # ssh
+
+ if [ $ENABLE_WEB_PORTS = "true" ]; then
+ ec2-authorize $TOOL_OPTS $CLUSTER_MASTER -p 50070 # NameNode web interface
+ ec2-authorize $TOOL_OPTS $CLUSTER_MASTER -p 50075 # DataNode web interface
+ ec2-authorize $TOOL_OPTS $CLUSTER_MASTER -p 60010 # HBase master web interface
+ ec2-authorize $TOOL_OPTS $CLUSTER_MASTER -p 60030 # HBase region server web interface
+ fi
+else
+ echo "Security group $CLUSTER_MASTER exists, ok"
+fi
+
+ec2-describe-group $TOOL_OPTS | egrep "[[:space:]]$CLUSTER[[:space:]]" > /dev/null
+if [ ! $? -eq 0 ]; then
+ echo "Creating group $CLUSTER"
+ ec2-add-group $TOOL_OPTS $CLUSTER -d "Group for HBase Slaves."
+ ec2-authorize $TOOL_OPTS $CLUSTER -o $CLUSTER -u $AWS_ACCOUNT_ID
+ ec2-authorize $TOOL_OPTS $CLUSTER -p 22 # ssh
+
+ if [ $ENABLE_WEB_PORTS = "true" ]; then
+ ec2-authorize $TOOL_OPTS $CLUSTER -p 50070 # NameNode web interface
+ ec2-authorize $TOOL_OPTS $CLUSTER -p 50075 # DataNode web interface
+ ec2-authorize $TOOL_OPTS $CLUSTER -p 60010 # HBase master web interface
+ ec2-authorize $TOOL_OPTS $CLUSTER -p 60030 # HBase region server web interface
+ fi
+
+ ec2-authorize $TOOL_OPTS $CLUSTER_MASTER -o $CLUSTER -u $AWS_ACCOUNT_ID
+ ec2-authorize $TOOL_OPTS $CLUSTER -o $CLUSTER_MASTER -u $AWS_ACCOUNT_ID
+else
+ echo "Security group $CLUSTER exists, ok"
+fi
+
+# Set up zookeeper group
+
+ec2-describe-group $TOOL_OPTS | egrep "[[:space:]]$CLUSTER_ZOOKEEPER[[:space:]]" > /dev/null
+if [ ! $? -eq 0 ]; then
+ echo "Creating group $CLUSTER_ZOOKEEPER"
+ ec2-add-group $TOOL_OPTS $CLUSTER_ZOOKEEPER -d "Group for HBase Zookeeper quorum."
+ ec2-authorize $TOOL_OPTS $CLUSTER_ZOOKEEPER -o $CLUSTER_ZOOKEEPER -u $AWS_ACCOUNT_ID
+ ec2-authorize $TOOL_OPTS $CLUSTER_ZOOKEEPER -p 22 # ssh
+
+ ec2-authorize $TOOL_OPTS $CLUSTER_MASTER -o $CLUSTER_ZOOKEEPER -u $AWS_ACCOUNT_ID
+ ec2-authorize $TOOL_OPTS $CLUSTER_ZOOKEEPER -o $CLUSTER_MASTER -u $AWS_ACCOUNT_ID
+ ec2-authorize $TOOL_OPTS $CLUSTER -o $CLUSTER_ZOOKEEPER -u $AWS_ACCOUNT_ID
+ ec2-authorize $TOOL_OPTS $CLUSTER_ZOOKEEPER -o $CLUSTER -u $AWS_ACCOUNT_ID
+else
+ echo "Security group $CLUSTER_ZOOKEEPER exists, ok"
+fi
+
63 bin/launch-hbase-cluster
@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Launch an EC2 cluster of HBase instances.
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+if [ -z $1 ]; then
+ echo "Cluster name required!"
+ exit 1
+fi
+CLUSTER=$1
+
+if [ -z $2 ]; then
+ echo "Must specify the number of slaves to start."
+ exit 1
+fi
+SLAVES=$2
+
+if [ -z $3 ]; then
+ echo "Must specify the number of zookeepers to start."
+ exit 1
+fi
+ZOOS=$3
+
+# Set up security groups
+
+if ! "$bin"/init-hbase-cluster-secgroups $CLUSTER ; then
+ exit $?
+fi
+
+# Launch the ZK quorum peers
+
+if ! "$bin"/launch-hbase-zookeeper $CLUSTER $ZOOS ; then
+ exit $?
+fi
+
+# Launch the HBase master
+
+if ! "$bin"/launch-hbase-master $CLUSTER ; then
+ exit $?
+fi
+
+# Launch the HBase slaves
+
+if ! "$bin"/launch-hbase-slaves $CLUSTER $SLAVES ; then
+ exit $?
+fi
96 bin/launch-hbase-master
@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Launch an EC2 HBase master.
+
+if [ -z $1 ]; then
+ echo "Cluster name required!"
+ exit 1
+fi
+
+CLUSTER=$1
+
+# Import variables
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+. "$bin"/hbase-ec2-env.sh
+
+if [ -z $AWS_ACCOUNT_ID ]; then
+ echo "Please set AWS_ACCOUNT_ID in $bin/hbase-ec2-env.sh."
+ exit 1
+fi
+
+type=$MASTER_INSTANCE_TYPE
+[ -z "$type" ] && type=$SLAVE_INSTANCE_TYPE
+arch=$MASTER_ARCH
+[ -z "$arch" ] && arch=$SLAVE_ARCH
+
+echo "Testing for existing master in group: $CLUSTER"
+MASTER_EC2_HOST=`ec2-describe-instances $TOOL_OPTS | awk '"RESERVATION" == $1 && "'$CLUSTER_MASTER'" == $4, "RESERVATION" == $1 && "'$CLUSTER_MASTER'" != $4'`
+MASTER_EC2_HOST=`echo "$MASTER_EC2_HOST" | awk '"INSTANCE" == $1 && "running" == $6 {print $4}'`
+
+if [ ! -z "$MASTER_EC2_HOST" ]; then
+ echo "Master already running on: $MASTER_EC2_HOST"
+ MASTER_HOST=`ec2-describe-instances $TOOL_OPTS $INSTANCE | grep INSTANCE | grep running | grep $MASTER_EC2_HOST | awk '{print $5}'`
+ echo $MASTER_HOST > $MASTER_PRIVATE_IP_PATH
+ echo $MASTER_EC2_HOST > $MASTER_IP_PATH
+ exit 0
+fi
+
+# Finding HBase image
+[ -z "$AMI_IMAGE" ] && AMI_IMAGE=`ec2-describe-images $TOOL_OPTS -a | grep -e $S3_BUCKET -e $S3_ACCOUNT | grep $HBASE_VERSION | grep $arch | grep available | awk '{print $2}'`
+
+# Start a master
+echo "Starting master with AMI $AMI_IMAGE (arch $arch)"
+# Substituting zookeeper quorum
+ZOOKEEPER_QUORUM=`cat $ZOOKEEPER_QUORUM_PATH`
+sed -e "s|%ZOOKEEPER_QUORUM%|$ZOOKEEPER_QUORUM|" \
+ -e "s|%EXTRA_PACKAGES%|$EXTRA_PACKAGES|" \
+ "$bin"/$USER_DATA_FILE > "$bin"/$USER_DATA_FILE.master
+INSTANCE=`ec2-run-instances $AMI_IMAGE $TOOL_OPTS -n 1 -g $CLUSTER_MASTER -k root -f "$bin"/$USER_DATA_FILE.master -t $type | grep INSTANCE | awk '{print $2}'`
+echo -n "Waiting for instance $INSTANCE to start"
+while true; do
+ printf "."
+ # get private dns
+ MASTER_HOST=`ec2-describe-instances $TOOL_OPTS $INSTANCE | grep running | awk '{print $5}'`
+ if [ ! -z $MASTER_HOST ]; then
+ echo " Started as $MASTER_HOST"
+ break;
+ fi
+ sleep 1
+done
+rm -f "$bin"/$USER_DATA_FILE.master
+
+MASTER_EC2_HOST=`ec2-describe-instances $TOOL_OPTS $INSTANCE | grep INSTANCE | grep running | grep $MASTER_HOST | awk '{print $4}'`
+echo $MASTER_HOST > $MASTER_PRIVATE_IP_PATH
+echo $MASTER_EC2_HOST > $MASTER_IP_PATH
+MASTER_EC2_ZONE=`ec2-describe-instances $TOOL_OPTS $INSTANCE | grep INSTANCE | grep running | grep $MASTER_HOST | awk '{print $11}'`
+echo $MASTER_EC2_ZONE > $MASTER_ZONE_PATH
+
+while true; do
+ REPLY=`ssh $SSH_OPTS "root@$MASTER_EC2_HOST" 'echo "hello"'`
+ if [ ! -z $REPLY ]; then
+ break;
+ fi
+ sleep 5
+done
+
+scp $SSH_OPTS $EC2_ROOT_SSH_KEY "root@$MASTER_EC2_HOST:/root/.ssh/id_rsa"
+ssh $SSH_OPTS "root@$MASTER_EC2_HOST" "chmod 600 /root/.ssh/id_rsa"
+
+MASTER_IP=`dig +short $MASTER_EC2_HOST`
+echo "Master is $MASTER_EC2_HOST, ip is $MASTER_IP, zone is $MASTER_EC2_ZONE."
60 bin/launch-hbase-slaves
@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Launch an EC2 HBase slaves.
+
+if [ -z $1 ]; then
+ echo "Cluster name required!"
+ exit 1
+fi
+
+CLUSTER=$1
+
+if [ -z $2 ]; then
+ echo "Must specify the number of slaves to start."
+ exit 1
+fi
+
+NO_INSTANCES=$2
+
+# Import variables
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+. "$bin"/hbase-ec2-env.sh
+
+if [ ! -f $MASTER_IP_PATH ]; then
+ echo "Must start Cluster Master first!"
+ exit 1
+fi
+
+[ -z "$AMI_IMAGE" ] && AMI_IMAGE=`ec2-describe-images $TOOL_OPTS -a | grep -e $S3_BUCKET -e $S3_ACCOUNT | grep $HBASE_VERSION | grep $SLAVE_ARCH | grep available | awk '{print $2}'`
+
+MASTER_HOST=`cat $MASTER_PRIVATE_IP_PATH`
+MASTER_ZONE=`cat $MASTER_ZONE_PATH`
+ZOOKEEPER_QUORUM=`cat $ZOOKEEPER_QUORUM_PATH`
+
+# Substituting master hostname and zookeeper quorum
+sed -e "s|%MASTER_HOST%|$MASTER_HOST|" \
+ -e "s|%ZOOKEEPER_QUORUM%|$ZOOKEEPER_QUORUM|" \
+ -e "s|%EXTRA_PACKAGES%|$EXTRA_PACKAGES|" \
+ "$bin"/$USER_DATA_FILE > "$bin"/$USER_DATA_FILE.slave
+
+# Start slaves
+echo "Starting $NO_INSTANCES AMI(s) with ID $AMI_IMAGE (arch $SLAVE_ARCH) in group $CLUSTER in zone $MASTER_ZONE"
+ec2-run-instances $AMI_IMAGE $TOOL_OPTS -n "$NO_INSTANCES" -g "$CLUSTER" -k root -f "$bin"/$USER_DATA_FILE.slave -t "$SLAVE_INSTANCE_TYPE" -z "$MASTER_ZONE" | grep INSTANCE | awk '{print $2}'
+
+rm "$bin"/$USER_DATA_FILE.slave
87 bin/launch-hbase-zookeeper
@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Launch the EC2 HBase Zookeepers.
+
+if [ -z $1 ]; then
+ echo "Cluster name required!"
+ exit 1
+fi
+
+if [ -z $2 ]; then
+ echo "Must specify the number of zookeeper quorum peers to start."
+ exit 1
+fi
+
+CLUSTER=$1
+NO_INSTANCES=$2
+
+# Import variables
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+. "$bin"/hbase-ec2-env.sh
+
+type=$ZOO_INSTANCE_TYPE
+[ -z "$type" ] && type=$SLAVE_INSTANCE_TYPE
+arch=$ZOO_ARCH
+[ -z "$arch" ] && arch=$SLAVE_ARCH
+
+# Finding HBase image
+[ -z "$ZOO_AMI_IMAGE" ] && ZOO_AMI_IMAGE=`ec2-describe-images $TOOL_OPTS -a | grep -e $S3_BUCKET -e $S3_ACCOUNT | grep hbase | grep $HBASE_VERSION | grep $arch | grep available | awk '{print $2}'`
+
+# Start Zookeeper instances
+
+echo "Starting ZooKeeper quorum ensemble."
+
+peers=""
+public_names=""
+inst=0
+while [ $inst -lt $NO_INSTANCES ] ; do
+ echo "Starting an AMI with ID $ZOO_AMI_IMAGE (arch $arch) in group $CLUSTER_ZOOKEEPER"
+ INSTANCE=`ec2-run-instances $ZOO_AMI_IMAGE $TOOL_OPTS -n 1 -g $CLUSTER_ZOOKEEPER -k root -t $type | grep INSTANCE | awk '{print $2}'`
+ echo -n "Waiting for instance $INSTANCE to start: "
+ while true; do
+ printf "."
+ # get private dns
+ priv=`ec2-describe-instances $TOOL_OPTS $INSTANCE | grep running | awk '{print $5}'`
+ if [ ! -z $priv ]; then
+ echo " Started ZooKeeper instance $INSTANCE as ${priv}"
+ break
+ fi
+ sleep 1
+ done
+ peers="$peers $priv"
+ public=`ec2-describe-instances $TOOL_OPTS $INSTANCE | grep running | awk '{print $4}'`
+ echo " Public DNS name is $public."
+ public_names="$public_names $public"
+ inst=$(($inst + 1))
+done
+
+ZOOKEEPER_QUORUM=`echo $peers | sed -e 's/ /,/g'`
+echo $ZOOKEEPER_QUORUM > $ZOOKEEPER_QUORUM_PATH
+echo "ZooKeeper quorum is $ZOOKEEPER_QUORUM."
+
+# Start Zookeeper quorum
+
+sleep 10
+echo "Initializing the ZooKeeper quorum ensemble."
+
+for host in $public_names ; do
+ echo " $host"
+ scp $SSH_OPTS "$bin"/hbase-ec2-init-zookeeper-remote.sh "root@${host}:/var/tmp"
+ ssh $SSH_OPTS "root@${host}" "sh -c \"ZOOKEEPER_QUORUM=\"$ZOOKEEPER_QUORUM\" sh /var/tmp/hbase-ec2-init-zookeeper-remote.sh\""
+done
31 bin/list-hbase-clusters
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# List running clusters.
+
+# Import variables
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+. "$bin"/hbase-ec2-env.sh
+
+# Finding HBase clusters
+CLUSTERS=`ec2-describe-instances $TOOL_OPTS | awk '"RESERVATION" == $1 && $4 ~ /-master$/, "INSTANCE" == $1' | tr '\n' '\t' | grep running | cut -f4 | rev | cut -d'-' -f2- | rev`
+
+[ -z "$CLUSTERS" ] && echo "No running clusters." && exit 0
+
+echo "Running HBase clusters:"
+echo "$CLUSTERS"
68 bin/revoke-hbase-cluster-secgroups
@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Clean up security groups for the EC2 HBase cluster
+
+if [ -z $1 ]; then
+ echo "Cluster name required!"
+ exit 1
+fi
+
+CLUSTER=$1
+
+# Import variables
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+. "$bin"/hbase-ec2-env.sh
+
+echo "Revoking security groups"
+
+ec2-describe-group $TOOL_OPTS | egrep "[[:space:]]$CLUSTER_MASTER[[:space:]]" > /dev/null
+if [ $? -eq 0 ]; then
+ ec2-revoke $TOOL_OPTS $CLUSTER_MASTER -o $CLUSTER_MASTER -u $AWS_ACCOUNT_ID
+fi
+
+ec2-describe-group $TOOL_OPTS | egrep "[[:space:]]$CLUSTER[[:space:]]" > /dev/null
+if [ $? -eq 0 ]; then
+ ec2-revoke $TOOL_OPTS $CLUSTER -o $CLUSTER -u $AWS_ACCOUNT_ID
+ ec2-revoke $TOOL_OPTS $CLUSTER_MASTER -o $CLUSTER -u $AWS_ACCOUNT_ID
+ ec2-revoke $TOOL_OPTS $CLUSTER -o $CLUSTER_MASTER -u $AWS_ACCOUNT_ID
+fi
+
+ec2-describe-group $TOOL_OPTS | egrep "[[:space:]]$CLUSTER_ZOOKEEPER[[:space:]]" > /dev/null
+if [ $? -eq 0 ]; then
+ ec2-revoke $TOOL_OPTS $CLUSTER_ZOOKEEPER -o $CLUSTER_ZOOKEEPER -u $AWS_ACCOUNT_ID
+ ec2-revoke $TOOL_OPTS $CLUSTER_MASTER -o $CLUSTER_ZOOKEEPER -u $AWS_ACCOUNT_ID
+ ec2-revoke $TOOL_OPTS $CLUSTER_ZOOKEEPER -o $CLUSTER_MASTER -u $AWS_ACCOUNT_ID
+ ec2-revoke $TOOL_OPTS $CLUSTER -o $CLUSTER_ZOOKEEPER -u $AWS_ACCOUNT_ID
+ ec2-revoke $TOOL_OPTS $CLUSTER_ZOOKEEPER -o $CLUSTER -u $AWS_ACCOUNT_ID
+fi
+
+ec2-describe-group $TOOL_OPTS | egrep "[[:space:]]$CLUSTER_MASTER[[:space:]]" > /dev/null
+if [ $? -eq 0 ]; then
+ ec2-delete-group $TOOL_OPTS $CLUSTER_MASTER
+fi
+
+ec2-describe-group $TOOL_OPTS | egrep "[[:space:]]$CLUSTER_ZOOKEEPER[[:space:]]" > /dev/null
+if [ $? -eq 0 ]; then
+ ec2-delete-group $TOOL_OPTS $CLUSTER_ZOOKEEPER
+fi
+
+ec2-describe-group $TOOL_OPTS | egrep "[[:space:]]$CLUSTER[[:space:]]" > /dev/null
+if [ $? -eq 0 ]; then
+ ec2-delete-group $TOOL_OPTS $CLUSTER
+fi
46 bin/terminate-hbase-cluster
@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Terminate a cluster.
+
+if [ -z $1 ]; then
+ echo "Cluster name required!"
+ exit 1
+fi
+
+CLUSTER=$1
+
+# Import variables
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+. "$bin"/hbase-ec2-env.sh
+
+# Finding HBase image
+HBASE_INSTANCES=`ec2-describe-instances $TOOL_OPTS | awk '"RESERVATION" == $1 && ("'$CLUSTER'" == $4 || "'$CLUSTER_MASTER'" == $4 || "'$CLUSTER_ZOOKEEPER'" == $4), "RESERVATION" == $1 && ("'$CLUSTER'" != $4 && "'$CLUSTER_MASTER'" != $4 && "'$CLUSTER_ZOOKEEPER'" != $4)'`
+HBASE_INSTANCES=`echo "$HBASE_INSTANCES" | grep INSTANCE | grep running`
+
+[ -z "$HBASE_INSTANCES" ] && echo "No running instances in cluster $CLUSTER." && exit 0
+
+echo "Running HBase instances:"
+echo "$HBASE_INSTANCES"
+read -p "Terminate all instances? [yes or no]: " answer
+
+if [ "$answer" != "yes" ]; then
+ exit 1
+fi
+
+ec2-terminate-instances $TOOL_OPTS `echo "$HBASE_INSTANCES" | awk '{print $2}'`

0 comments on commit 7351901

Please sign in to comment.