Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

Blog post for deploying drizzle on EC2 with chef.

  • Loading branch information...
commit 4621415138e638841c9bc6b8b9186d642dfeee32 1 parent fa04386
@posulliv authored
View
294 _posts/drizzle/2011-04-07-drizzle-chef.html
@@ -0,0 +1,294 @@
+---
+layout: post
+title: Deploy Drizzle on EC2 with chef
+---
+<p>
+This post is a tutorial on how to deploy Drizzle on an EC2 instance using chef and the <a href="http://www.opscode.com/platform/">Opscode Chef</a> platform. The tutorial is specifically targetted at Ubuntu platforms. In particular, the procedures outlined here have only been tested on Ubuntu 10.04. It is expected however that the instructions here should apply on other Ubuntu versions with minimal modifications needed.
+</p>
+
+<h2>The Opscode Platform</h2>
+
+<p>
+In this article, we'll use the Opscode platform since it provides an easy way for anyone to get started with chef. If you are a new user, proceed to <a href="https://community.opscode.com/users/new">sign up</a> for a new account. Once you are signed up, the next step is to create a new organization. For this article, I'm going to create an organization named drizzle-test. Once your organization is created, you should see the organization in your list of organizations when you click on the Organizations link at the top right of the opscode console. My view looks like (you should be able to click on the image to see a larger version):
+</p>
+
+<img src="../../../images/console_orgs.png" width=750 />
+
+<h2>Configure AWS</h2>
+
+<p>
+An assumption made in this article is that you have an <a href="http://aws.amazon.com/">AWS</a> account. If you don't, signing up is relatively straightforward.
+</p>
+
+<p>
+There are a few items that need to be configured for EC2 that we need to do to make our lives easier before starting with chef. Amazon blocks all incoming traffic to EC2 instances by default. SSH is used by chef to access and bootstrap a newly created instance. We want to allow SSH traffic to our EC2 instances and for this article, I want to permit traffic to the drizzle port (default drizzle port is 4427) as well. This is accomplished using the AWS console. We need to configure Security Groups. You can either create a new security group and modify the default security group. For this article, I'll create a new security group named drizzle and add the appropriate rules. After creating the group and adding the rules, the security group details should look like:
+</p>
+
+<img src="../../../images/security_group.png" width=750 />
+
+<p>
+I'll also create a new key pair in the AWS console specifically for this article. I'm going to give this key pair the name drizzle. After creating the key pair, I copy the downloaded private key to my SSH folder and update permissions on the key:
+</p>
+
+<pre>
+mv ~/Downloads/drizzle.pem ~/.ssh/
+chmod 600 ~/.ssh/drizzle.pem
+</pre>
+
+<h2>Install chef</h2>
+
+<p>
+To install chef on Ubuntu is quite straightforward. Opscode maintains an APT repository which I simply need to add to my sources list. In the file <code>/etc/apt/sources.list.d/opscode.list</code>, add (and replace lucid with whatever release you are running):
+</p>
+
+<pre>
+deb http://apt.opscode.com/ lucid main
+</pre>
+
+<p>
+Next, I need to add the GPG key:
+</p>
+
+<pre>
+wget -qO - http://apt.opscode.com/packages@opscode.com.gpg.key | sudo apt-key add -
+sudo apt-get update
+</pre>
+
+<p>
+To install chef, its as simple as installing the chef package:
+</p>
+
+<pre>
+sudo apt-get install chef
+</pre>
+
+<p>
+When prompted for the server URL during this package installation, you can leave it blank. We will be configuring this later. You can also stop and disable the chef-client service now if you wish since we will only be using the <code>knife</code> utility in this article. Finally, verify the version you have installed:
+</p>
+
+<pre>
+knife -v
+</pre>
+
+<p>
+For this article, the output of the above command needs to be a least 0.9.14
+</p>
+
+<p>
+Other packages required for this article are rubygems and git:
+</p>
+
+<pre>
+sudo apt-get install rubygems git
+</pre>
+
+<p>
+Once rubygems is installed, there a few gems required for interacting with EC2:
+</p>
+
+<pre>
+sudo gem install net-ssh net-ssh-multi fog highline
+</pre>
+
+<h2>Configure chef</h2>
+
+<p>
+We are now all set to get started. The first thing to do is create a chef repository on your workstation. In this article, I will use git for this:
+</p>
+
+<pre>
+git clone https://github.com/opscode/chef-repo.git drizzle-chef-repo
+</pre>
+
+<p>
+Create a <code>.chef</code> directory within this repository. This directory contains all the configuration files for <b>just this</b> repository:
+</p>
+
+<pre>
+mkdir -p ~/drizzle-chef-repo/.chef
+</pre>
+
+<p>
+Next, we need to download keys and knife configuration files from the Opscode platform that will be used for interacting with Opscode platform. Keys are needed for both your user and organization on the Opscode Platform. To retrieve your user key (if you did not download it when signing up), click on your username through the console and you will a 'get private key' link on your account page:
+</p>
+
+<img src="../../../images/user_key.png" width=750 />
+
+<p>
+After Downloading this key, I need to place it in the configuration directory for the chef repository I am using here:
+</p>
+
+<pre>
+mv ~/Downloads/posulliv.pem ~/drizzle-chef-repo/.chef
+</pre>
+
+<p>
+For your organization, click on the 'Regenerate validation key' link and 'Generate knife config' link from the organizations over page as mentioned in the first section of this article. After clicking those 2 links, you will have 2 files: 1) drizzle-test-validator.pem and 2) knife.rb. Move these 2 files into the configuration directory for the chef repository being used for this article:
+</p>
+
+<pre>
+mv ~/Downloads/drizzle-test-validator.pem ~/drizzle-chef-repo/.chef
+mv ~/Downloads/knife.rb ~/drizzle-chef-repo/.chef
+</pre>
+
+<p>
+From now on, whenever you are in the <code>drizzle-chef-repo</code> directory, the <code>knife</code> utility will connect to the Opscode Platform. To verify this, lets list out the current clients:
+</p>
+
+<pre>
+posulliv@curragh:~/drizzle-chef-repo$ knife client list
+[
+ "drizzle-test-validator"
+]
+posulliv@curragh:~/drizzle-chef-repo$
+</pre>
+
+<p>
+We need to tell <code>knife</code> about our AWS credentials. This is done by adding the following 2 lines to your <code>knife.rb</code> file in the <code>~/drizzle-chef-repo/.chef</code> directory:
+</p>
+
+<pre>
+knife[:aws_access_key_id] = "Your AWS Access Key"
+knife[:aws_secret_access_key] = "Your AWS Secret Access Key"
+</pre>
+
+<p>
+After adding these credentials I should now be able to list all the EC2 instances associated with my AWS account:
+</p>
+
+<pre>
+posulliv@curragh:~/drizzle-chef-repo$ knife ec2 server list
+Instance ID Public IP Private IP Flavor Image Security Groups State
+i-5e1ce433 50.17.249.89 10.253.30.159 m1.large ami-879f70ee AkibanWeb running
+i-1bcb4f77 50.16.188.89 10.112.233.119 t1.micro ami-548c783d AkibanWeb running
+i-d6fa10b9 50.17.34.183 10.243.14.15 m1.large ami-548c783d AkibanQA running
+i-98db31f7 50.16.137.154 10.114.246.151 m1.large ami-548c783d AkibanQA running
+i-1e16fc71 174.129.139.237 10.195.205.139 m1.large ami-548c783d AkibanQA running
+posulliv@curragh:~/drizzle-chef-repo$
+</pre>
+
+<h2>Drizzle Cookbook</h2>
+
+<p>
+chef should now be configured to work with your AWS account. The next step is to decide on what roles or recipes you want to apply to an instance you create. Since this article is on drizzle, I'll show how to bootstrap an EC2 instance with drizzle. I have developed a simple drizzle cookbook in a fork of Opscode's official cookbook repository that can be retrieved with git:
+</p>
+
+<pre>
+cd ~/drizzle-chef-repo
+rm -rf cookbooks
+git clone git://github.com/posulliv/cookbooks.git
+</pre>
+
+<p>
+I have opened a pull request for this fork to get merged into Opscode's official repository. Hopefully, it will get merged in soon.
+</p>
+
+<p>
+Now we want to upload cookbooks to our chef server. The only cookbook I will upload in this article is the Drizzle cookbook:
+</p>
+
+<pre>
+cd ~/drizzle-chef-repo
+knife cookbook upload drizzle
+</pre>
+
+It is simple to list the cookbooks that have been uploaded so far to your chef server:
+
+<pre>
+posulliv@curragh:~/drizzle-chef-repo$ knife cookbook list
+[
+ "drizzle"
+]
+posulliv@curragh:~/drizzle-chef-repo$
+</pre>
+
+<h2>Create and Verify EC2 Instance</h2>
+
+<p>
+We are now ready to create an EC2 instance and have it bootstrap itself and install the drizzle GA release! You will see a spew of output when you issue the command below (feel free to use any AMI image or flavor you wish, I just picked one):
+</p>
+
+<pre>
+knife ec2 server create "recipe[drizzle]" \
+--image ami-2d4aa444 \
+--flavor m1.small \
+--groups drizzle \
+--ssh-key drizzle \
+--identity-file ~/.ssh/drizzle.pem \
+--ssh-user ubuntu
+</pre>
+
+<p>
+To verify the server is created, first we check in the server list output from EC2:
+</p>
+
+<pre>
+posulliv@curragh:~/drizzle-chef-repo$ knife ec2 server list
+Instance ID Public IP Private IP Flavor Image Security Groups State
+i-5e1ce433 50.17.249.89 10.253.30.159 m1.large ami-879f70ee AkibanWeb running
+i-1bcb4f77 50.16.188.89 10.112.233.119 t1.micro ami-548c783d AkibanWeb running
+i-d6fa10b9 50.17.34.183 10.243.14.15 m1.large ami-548c783d AkibanQA running
+i-98db31f7 50.16.137.154 10.114.246.151 m1.large ami-548c783d AkibanQA running
+i-1e16fc71 174.129.139.237 10.195.205.139 m1.large ami-548c783d AkibanQA running
+i-c03b5caf 50.17.153.76 10.202.253.78 m1.small ami-2d4aa444 drizzle running
+posulliv@curragh:~/drizzle-chef-repo$
+</pre>
+
+<p>
+We should also verify that it is listed as a node:
+</p>
+
+<pre>
+posulliv@curragh:~/drizzle-chef-repo$ knife node list
+[
+ "i-c03b5caf"
+]
+posulliv@curragh:~/drizzle-chef-repo$
+</pre>
+
+<p>
+Finally, if I log onto the EC2 instance I should be able to connect to drizzle:
+</p>
+
+<pre>
+posulliv@curragh:~$ ssh -i ~/.ssh/drizzle.pem ubuntu@50.17.153.76
+Linux ip-10-116-210-131 2.6.32-305-ec2 #9-Ubuntu SMP Thu Apr 15 04:14:01 UTC 2010 i686 GNU/Linux
+Ubuntu 10.04 LTS
+
+Welcome to Ubuntu!
+ * Documentation: https://help.ubuntu.com/
+
+ System information as of Mon Apr 11 23:01:28 UTC 2011
+
+ System load: 0.36 Memory usage: 13% Processes: 55
+ Usage of /: 8.6% of 9.92GB Swap usage: 0% Users logged in: 0
+
+ Graph this data and manage this system at https://landscape.canonical.com/
+---------------------------------------------------------------------
+At the moment, only the core of the system is installed. To tune the
+system to your needs, you can choose to install one or more
+predefined collections of software by running the following
+command:
+
+ sudo tasksel --section server
+---------------------------------------------------------------------
+
+A newer build of the Ubuntu lucid server image is available.
+It is named 'release' and has build serial '20110201.1'.
+Last login: Mon Apr 11 22:27:04 2011 from 12.43.172.10
+ubuntu@ip-10-116-210-131:~$ drizzle
+Welcome to the Drizzle client.. Commands end with ; or \g.
+Your Drizzle connection id is 9
+Connection protocol: mysql
+Server version: 2011.03.13 Ubuntu
+
+Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
+
+drizzle>
+</pre>
+
+<h2>Conclusion</h2>
+
+<p>
+Hopefully, this tutorial proves useful. I hope to work more on the Drizzle cookbook in the near future and add support for the various plugin types present in Drizzle.
+</p>
View
BIN  images/console_orgs.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View
BIN  images/security_group.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View
BIN  images/user_key.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Please sign in to comment.
Something went wrong with that request. Please try again.