Skip to content
Find file
Fetching contributors…
Cannot retrieve contributors at this time
71 lines (48 sloc) 3.09 KB

Settings files and Credentials


You should already be familiar with hadoop and with the Amazon cloud.

Choose a name for your cluster, a domain name for your chef server, and a domain name for the cluster’s master node. (It’s OK if your chef server is the same as the master node, In this example, we’ll be using

  • ‘zaius’ as the cluster name
  • ‘’ as the hostname of the chef server
  • ‘’ as the hostname of the cluster’s master node

Install dependencies

Install these gems:

  • chef
  • configliere
  • amazon-ec2
  • infochimps-poolparty

NOTE: Please use the infochimps branch of poolparty for spot instance support and other tweaks

Create Settings Directory

  mkdir ~/.hadoop-ec2
  mkdir ~/.hadoop-ec2/keypairs ~/.hadoop-ec2/certs
  ln -nfs ~/.hadoop-ec2 ~/.poolparty
  ln -nfs ~/.hadoop-ec2 ~/.chef

From this code repo dir, copy the template config files over.

  cd PATH/TO/cluster_chef
  cp ./config/poolparty-example.yaml   ~/.hadoop-ec2/poolparty.yaml 
  ln -nfs ~/.hadoop-ec2/poolparty.yaml ~/.hadoop-ec2/aws
  # optional:
  ( cd ~/.hadoop-ec2 && git init && git add . && git commit -m "Initial commit" )

Go on the Amazon AWS Scavenger Hunt

  • Register for EC2, if you haven’t. You may have to click through a license agreement and check your email. If you want to use the Broham tool for assigning cluster roles, sign up for Simple

Sign in to the aws console and collect the following power-ups:

  • Allocate an elastic IP for the head node.
  • On your DNS server (or, set a fully-qualified domain name pointing to that IP address — say,
  • Create an EC2 keypair named ‘zaius’, and store its .pem file in ~/.hadoop-ec2/keypairs/zaius.pem
  • If you’ve never done so, store your EC2 certificate files in ~/.hadoop-ec2/certs/pk-WHATEVER.pem and ~/.hadoop-ec2/certs/cert-WHATEVER.pem.
  • Fix the permissions of your keypairs directory: chmod 600 ~/.hadoop-ec2/*/*.pem.
  • AWS Account ID: It’s near the top of the page, right below “Welcome, Whoever You Are”: something like “8675-3000-9999”. Record it without the dashes: “867530009999”
  • AWS Access Key: From the Your Account / Access Keys; 20 characters, something like “12345ABCDE12345ABCDE”.
  • AWS Secret Access Key: From the Your Account / Access Keys; 40 characters, something like “gobBBLEDy/goo+kANDlattersandnumbersd00dA”.

Settings files

In the file ~/.hadoop-ec2/poolparty.yaml,

  • Set the access_key, secret_access_key and account_id using the values found above.
  • Put the URL of your chef server in the top-level [:user_data][:chef] section; and again in the node_name for your chef server
  • Put the elastic IP you allocated in the section for your chef server
  • Update your preferred availability zone, etc

Next Steps

Next, follow the instructions for Step 2: Launch the Chef Server / Hadoop Master.

Jump to Line
Something went wrong with that request. Please try again.