Settings files and Credentials
You should already be familiar with hadoop and with the Amazon cloud.
Choose a name for your cluster, a domain name for your chef server, and a domain name for the cluster’s master node. (It’s OK if your chef server is the same as the master node, In this example, we’ll be using
- ‘zaius’ as the cluster name
- ‘chef.yourdomain.com’ as the hostname of the chef server
- ‘zaius.yourdomain.com’ as the hostname of the cluster’s master node
Install these gems:
NOTE: Please use the infochimps branch of poolparty for spot instance support and other tweaks
Create Settings Directory
mkdir ~/.hadoop-ec2 mkdir ~/.hadoop-ec2/keypairs ~/.hadoop-ec2/certs ln -nfs ~/.hadoop-ec2 ~/.poolparty ln -nfs ~/.hadoop-ec2 ~/.chef
From this code repo dir, copy the template config files over.
cd PATH/TO/cluster_chef cp ./config/poolparty-example.yaml ~/.hadoop-ec2/poolparty.yaml ln -nfs ~/.hadoop-ec2/poolparty.yaml ~/.hadoop-ec2/aws # optional: ( cd ~/.hadoop-ec2 && git init && git add . && git commit -m "Initial commit" )
Go on the Amazon AWS Scavenger Hunt
- Register for EC2, if you haven’t. You may have to click through a license agreement and check your email. If you want to use the Broham tool for assigning cluster roles, sign up for Simple
Sign in to the aws console and collect the following power-ups:
- Allocate an elastic IP for the head node.
- On your DNS server (or http://dyndns.org), set a fully-qualified domain name pointing to that IP address — say, http://chef.yourdomain.com:4040
- Create an EC2 keypair named ‘zaius’, and store its .pem file in ~/.hadoop-ec2/keypairs/zaius.pem
- If you’ve never done so, store your EC2 certificate files in
- Fix the permissions of your keypairs directory:
chmod 600 ~/.hadoop-ec2/*/*.pem.
- AWS Account ID: It’s near the top of the page, right below “Welcome, Whoever You Are”: something like “8675-3000-9999”. Record it without the dashes: “867530009999”
- AWS Access Key: From the Your Account / Access Keys; 20 characters, something like “12345ABCDE12345ABCDE”.
- AWS Secret Access Key: From the Your Account / Access Keys; 40 characters, something like “gobBBLEDy/goo+kANDlattersandnumbersd00dA”.
In the file
- Set the
account_idusing the values found above.
- Put the URL of your chef server in the top-level [:user_data][:chef] section; and again in the node_name for your chef server
- Put the elastic IP you allocated in the section for your chef server
- Update your preferred availability zone, etc
Next, follow the instructions for Step 2: Launch the Chef Server / Hadoop Master.