Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
branch: master
Fetching contributors…

Cannot retrieve contributors at this time

125 lines (85 sloc) 4.842 kb

Option A: You want a combined Chef Server and Hadoop Master

If you are just getting started, this is probably the best option. If you already have a chef server, go with Option B.

To kick off the chef+master node, go to the cluster_chef directory and run:


  $ cd ~/path/to/cluster_chef
  $ cloud start -n chefmaster -c clouds/zaius_clouds.rb
  • If you’re using the west coast availability zone, first run export EC2_URL=https://us-west-1.ec2.amazonaws.com to work around a bug in the right_aws gem

You can check the AWS console for progress towards startup. If there are problems, see Tips and Trobleshooting.

Set up the chef WebUI (dashboard)

  • Visit the chef webui: http://chef.YOURDOMAIN.COM:4040 (note that the port number for the webui, the thing people use, is 4040. The port number for the chef server, the think knife and clients and other robots use, is 4000.)
  • Find the webui admin’s initial password: sudo cat /etc/chef/server.rb | grep -i pass.
  • Set the webui admin’s password to something secure.

Log in to your server

Log in to the chef server to retrieve a few settings and credentials you’ll need:


  cloud ssh -n master -c clouds/zaius_clouds.rb

Or alternatively,


  ssh -i ~/.hadoop-ec2/keypairs/zaius.pem ubuntu@ec2-123-45-67-89.compute-1.amazonaws.com

Set up knife

On the server, set up knife:


  sudo knife configure -i
  Your chef server URL? http://chef.YOURDOMAIN.com:4000
  Your client user name? knife_user
  Your validation client user name? chef-validator
  Path to a chef repository (or leave blank)? 
  WARN: Creating initial API user...

Option B: You already have a Chef Server

If you already have a chef server,

  • Set up a knife_user client: through the webui, or using the command line as above.
  • Find the server’s validation key, which is probably in /etc/chef/validation.pem.
  • Point the user_data:chef_server in ~/.hadoop-ec2/poolparty.yaml to your chef server url, eg “http://chef.YOURDOMAIN.com:4000”.

Set up your local environment for the chef server

The following applies whether you chose option A or option B.

Gather Chef credentials

From your chef server (however it was set up),

  • Copy the chef credentials over (Because of permissions, it’s probably easiest to do this by copy/pasting rather than scp.):
    • Copy ~/.chef/knife_user.pem on the server to ~/.hadoop-ec2/keypairs/knife_user.pem on your computer
    • Copy /etc/chef/validation.pem on the server to ~/.hadoop-ec2/keypairs/chef-validator.pem on your computer.
    • (note that the paths and names are different on your local machine)
  • Fix their permissions: chmod og-rwx ~/.hadoop-ec2/keypairs/*.pem
  • Copy the config/knife.rb file from this repo to ~/.chef/knife.rb. Check whether any settings in that file need tuning.

High on knife (checking that your local configuration works)

If everything went right, knife client list from your local machine should show something like


  [
    "chef-validator", 
    "chef-webui",
    "knife_user"
  ]

Import cookbooks, roles and databags to the server

Load the cookbooks, roles and recipes into the chef server. Chef-client will pull them from the chef server to populate its node.


  cd ~/path/to/cluster_chef
  knife cookbook upload --all
  for foo in roles/*.rb ; do echo $foo ; knife role from file $foo ; done
  rake load_data_bags
  rake load_data_bags

(run the rake load_data_bags task twice: once to create, again to populate each data bag)

Option B: You already have a Chef Server (part 2)

Once the chef server is populated, you can launch

To kick off the master node, go to the cluster_chef directory and run:


  cloud start -n master -c clouds/zaius_clouds.rb
  • If you’re using the west coast availability zone, first run export EC2_URL=https://us-west-1.ec2.amazonaws.com to work around a bug in the right_aws gem

You can check the AWS console for progress towards startup. If there are problems, see Tips and Trobleshooting.

Log in to your server

To log in to the cluster master:


  cloud ssh -n master -c clouds/zaius_clouds.rb

Or alternatively,


  ssh -i ~/.hadoop-ec2/keypairs/zaius.pem ubuntu@zaius.YOURDOMAIN.com

Next Steps

Next, follow the instructions for Step 3: Hadoop Master.

Jump to Line
Something went wrong with that request. Please try again.