Skip to content

Software Update Howto

Jump to bottom Edit New page
jbau edited this page Sep 18, 2012 · 9 revisions

Setup

  1. ssh to deploy.dev as yourself.
ssh -A deploy.dev.c2gops.com   # -A forwards your ssh keys
  1. Check out our repo. I like having mine just in the root of my homedir, but you can do whatever makes sense.
git clone https://github.com/Stanford-Online/class2go.git
  1. go to class2go/chef, run script that will create links to shared files that you need for environments and knife itself
sef@deploy (master) ~/class2go> cd chef
sef@deploy (master) ~/class2go/chef> ./setup_deploy_links.sh 
`environments/audit.rb' -> `/home/shared/class2go/chef/environments/audit.rb'
`environments/jenkins.rb' -> `/home/shared/class2go/chef/environments/jenkins.rb'
`environments/prod.rb' -> `/home/shared/class2go/chef/environments/prod.rb'
`environments/stage.rb' -> `/home/shared/class2go/chef/environments/stage.rb'
`environments/dev.rb' -> `/home/shared/class2go/chef/environments/dev.rb'
`.chef/knife.rb' -> `/home/shared/class2go/chef/.chef/knife.rb'
`.chef/class2go-validator.pem' -> `/home/shared/class2go/chef/.chef/class2go-validator.pem'
`.chef/sefk.pem' -> `/home/shared/class2go/chef/.chef/sefk.pem'
  1. Verify that you can use the knife command to talk to the Chef servers now. You shoudl get a list of server names.
sef@deploy (master) ~/class2go/chef> knife node list
  app1.audit
  app1.stage
  app2.stage
  util1.audit
  util1.dev
  util1.stage

Updating Servers

The easiest way to update servers is using the knife command to run chef client.

Unfortunately we are using different images for our util and app servers, so we can't use one command to drive both. We need to specify "-x bitnami" for the app servers, and "-x ubuntu" for the util servers.

Two important steps before pushing anything.

  1. Upload Cookbooks Chef downloads these from the server, not from your local repo, so if someone last deployed from another branch, you could end up with the wrong set of cookbooks. So:

    cd class2go/chef
knife cookbook upload -a

  2. Reload Environments This doesn't have to happen every time, but if you believe that there is a chagne to anything in the environment files, you need to re-upload

    cd class2go/chef/environments 
knife environment from file stage.rb

Make sure that you can use knife to hit all servers of a kind. This searches for all nodes named app##.stage and runs the hostname command on them.

sef@deploy (master) ~/class2go/chef> knife ssh "name:app*.stage" -x
bitnami hostname 
ec2-54-245-17-236.us-west-2.compute.amazonaws.com app1.stage.c2gops.com
ec2-50-112-237-200.us-west-2.compute.amazonaws.com app2.stage.c2gops.com

If this works then run chef-client on all the machines. This will pull code and update config as needed.

cd class2go/chef
knife ssh "name:app*.stage" -x bitnami 'sudo chef-client'

Installing Servers from Scratch

Chef stores keys for servers that are tied to that server's name. Creating a new server, eg. app100.stage, is fine, but to create a new server to re-use an existing name there are three additional steps:

  1. On the Chef Server console (http://manage.opscode.com/) and delete the node and the client. The knife create command will fail if this isn't done first.

  2. On the AWS EC2 console, delete the old instance with that name. Note this doesn't have to be done before the knife ec2 create command, you'll just end up with two instances with the same name, be careful to stop/terminate the right one! Caveat emptor.

  3. Install the machine. The commands (per machine, per environment) can be found in the Setup Notes: EC2 wiki page. For example, this is the command for a staging utility machine:

    knife ec2 server create -I ami-4438b474 -x ubuntu -N util1.stage --region us-west-2 -Z us-west-2b -f m1.small -G stage -E stage -r 'role[util]'

Weekly Software Install

Database

  1. take database backup, eg prod-pre-sprint-12

Load Balancer

  1. EC2: make sure the prod-first has just app1.prod in it, and that it is in service

  2. EC2: take app1 out of normal prod load balancer

  3. Route53: have www.class.stanford.edu point to prod-first load balancer

  4. Verify

dig www.class.stanford.edu host prod-first-1481346004.us-west-2.elb.amazonaws.com

IP's should match up

  1. Browser http://www.client.stanford.edu should work now before sw upgrade.

Upgrade prod-first

  1. deploy, as you: Prepare

    • tmux (forward ssh agent?)
    • sync branch to production
    • rake install
    • knife environment from file prod.rb
    • knife ssh 'name:app1.prod' -x bitnami hostname

  2. ** deploy, as you**: Install

    • log in IRC
    • knife ssh 'name:app1.prod' -x bitnami 'sudo chef-client'

  3. app1, bitnami make sure branch is sane

    • git log --color --oneline --decorate
    • (If need be) git reset --hard HEAD

Basic functional checklist on http://www.class.stanford.edu/

  1. Post announcement

  2. Visit forum

Production Load

  1. re-add app1.prod to prod load balancer

  2. tail access-log and error-log

Upgrade other two machines

If doing a major upgrade, do one at a time

  1. take machine out of prod load balancer
  2. knife ssh 'name:app2.prod' -x bitnami 'sudo chef-client'
  3. (any special steps)
  4. re-add to prod load balancer

Back to normal

  1. EC2: prod load balancer has three machines

  2. Route53: www.class.stanford.edu points to prod load balancer (not prod-first)

Add a custom footer
Add a custom sidebar
Clone this wiki locally