Refactor deployment #377

Closed
wants to merge 28 commits into
from

Conversation

Projects
None yet
4 participants
Contributor

CloCkWeRX commented Jun 9, 2014

This should fix #373 (and #339)

  • Use rbenv
  • Fixes for Centos

I didn't overly test the development deploy.

Wiki documentation would need to be updated slightly.

{

"name" : "huginn_production",

"chef_type" : "role",

"json_class" : "Chef::Role",

"description" : "Huginn Production Environment",

"default_attributes" : {
  "mysql": {
    "server_root_password": "password",
    "server_repl_password": "",
    "server_debian_password": ""
  },
    "nginx" : {
       "init_style" : "upstart"
    }
},

"run_list":[
             "recipe[git]",
             "recipe[apt]",
             "recipe[mysql::server]",
             "recipe[nodejs::install_from_binary]",
             "recipe[nginx]",
             "recipe[ruby_build]",
             "recipe[rbenv]",
             "recipe[huginn::ruby]",
             "recipe[huginn::user]",
             "recipe[huginn::production]"
           ]
}

The roles now share most of the code for setup, with only the huginn::production and huginn::development recipes differing.

You can now pick your own attributes, like ruby version, git repo, etc:

default['rbenv']['ruby_version'] = "2.1.2"
default['rbenv']['plugins'] = [{
  'name' => 'rbenv-sudo',
  'git_url' => 'git://github.com/dcarley/rbenv-sudo.git'
}]

default['huginn']['repo'] = "https://github.com/cantino/huginn.git"
default['huginn']['branch'] = "master"
default['huginn']['rails_env'] = "production"
default['huginn']['keep_releases'] = 5
default['huginn']['user'] = "huginn"
default['huginn']['group'] = "huginn"
default['huginn']['deploy_path'] = "/home/#{node['huginn']['user']}"
default['huginn']['env']['invitation_code'] = "try-huginn-secretly"

Mail details still probably need refactoring.

Contributor

CloCkWeRX commented Jun 9, 2014

@alias1 can I get you to check this on ubuntu/centos? It works for me on one instance of centos 6.4

Coverage Status

Coverage remained the same when pulling 71408ed on CloCkWeRX:refactor_deployment into e48938b on cantino:master.

Owner

cantino commented Jun 9, 2014

@dsander and @racktear should also look at this. They've been doing most of the Chef stuff.

Member

0xdevalias commented Jun 11, 2014

Related: #339

Don't have time to look through this for now, but if it matches up with all the stuff I was trying to do, and solves the issues I was having, I would consider this as completing the bounty I set on the other one.

Will hopefully check it out later today or soon at least.

(Edit: Also, tagging #334 as once this is handled properly I think that one will be far easier to deal with. Let chef handle the actual system setup, but be able to do the raw deploy with capistrano)

Member

0xdevalias commented Jun 18, 2014

Still definitely intending on playing with this, but given things in life it probably isn't going to be till next week now (5+ days from now)

+source 'https://rubygems.org'
+
+gem 'librarian-chef'
+gem 'knife-solo'
@0xdevalias

0xdevalias Jun 23, 2014

Member

New line at end of file?

@0xdevalias

0xdevalias Jun 23, 2014

Member

Also, @cantino said on my branch that these should be commented out with a note if they're included in the Gemfile, as not everyone will want/need them for a normal install.

@@ -8,3 +8,5 @@ cookbook 'git', :git => 'git://github.com/opscode-cookbooks/git.git'
cookbook 'nginx', :git => 'git://github.com/opscode-cookbooks/nginx.git'
cookbook 'mysql', :git => 'git://github.com/opscode-cookbooks/mysql.git'
cookbook 'nodejs', :git => 'git://github.com/mdxp/nodejs-cookbook.git'
+cookbook 'rbenv', :git => 'git://github.com/fnichol/chef-rbenv.git', :ref => 'v0.7.2'
+cookbook 'ruby_build'
@0xdevalias

0xdevalias Jun 23, 2014

Member

Have you seen cookbook 'sudo', :git => 'git://github.com/opscode-cookbooks/sudo.git' ?

@CloCkWeRX

CloCkWeRX Jun 23, 2014

Contributor

I have, but it was honestly a lot simpler to do

group "sudo"

group "sudo" do
   action :modify
   members "huginn"
   append true
end

Happy to re-look at it if there are compelling advantages; or this doesn't work on various operating systems.

@@ -25,6 +25,9 @@
"recipe[mysql::server]",
"recipe[mysql::client]",
"recipe[nodejs::install_from_binary]",
- "recipe[huginn_development]"
+ "recipe[ruby_build]",
@0xdevalias

0xdevalias Jun 23, 2014

Member

No "recipe[rbenv]", ?

@CloCkWeRX

CloCkWeRX Jun 23, 2014

Contributor

Covered with include_recipe "rbenv::system" in huginn::ruby, via a depends.

The main reason for this being present is so the rbenv build actions in huginn::ruby are available.
See https://github.com/fnichol/chef-rbenv#-rubies

@0xdevalias

0xdevalias Jun 23, 2014

Member

Does that mean that it's also not required in the huginn_production.json?

@CloCkWeRX

CloCkWeRX Jun 23, 2014

Contributor

Should be the case.

@@ -0,0 +1,10 @@
+default['rbenv']['ruby_version'] = "1.9.3-p353"
@0xdevalias

0xdevalias Jun 23, 2014

Member

Should this be more like 2.1.2?

At the very least (if sticking to 1.9.3) 1.9.3-p547

@CloCkWeRX

CloCkWeRX Jun 23, 2014

Contributor

Went to p547. I would go to 2.1.2, just the other various documentation and syntax still say 1.9.3. I'm personally running 2.1.2.
Perhaps worth a separate PR/bug/ etc?

@0xdevalias

0xdevalias Jun 23, 2014

Member

nods Happy to just leave it at the latest 1.9.3 for now :)

+default['huginn']['user'] = "huginn"
+default['huginn']['group'] = "huginn"
+default['huginn']['deploy_path'] = "/home/#{node['huginn']['user']}"
+default['huginn']['env']['invitation_code'] = "try-huginn-secretly"
@0xdevalias

0xdevalias Jun 23, 2014

Member

Newline at end of file

@0xdevalias

0xdevalias Jun 23, 2014

Member

Also, since you use the group 'sudo' elsewhere, won't these defaults need to be +default['huginn']['group'] = "huginn"

And in that case, shouldn't you be using the default options instead of hardcoded strings in the user creation stuff?

@@ -0,0 +1,2 @@
+web: rbenv sudo bundle exec unicorn_rails -c config/unicorn.rb -E production
+jobs: rbenv sudo RAILS_ENV=production bundle exec rails runner bin/threaded.rb
@0xdevalias

0xdevalias Jun 23, 2014

Member

Newline at end of file

+ user "huginn"
+end
+
+bash "huginn dependencies" do
@0xdevalias

0xdevalias Jun 23, 2014

Member

Similarly here, should you be using +default['huginn']['user'] instead of the hardcoded?

+ export LC_ALL="en_US.UTF-8"
+ sudo bundle install
+ sed s/REPLACE_ME_NOW\!/$(sudo bundle exec rake secret)/ .env.example > .env
+ sudo bundle exec rake db:create
@0xdevalias

0xdevalias Jun 23, 2014

Member

Also, should look at using the creates file guard to prevent repeating db:create, db:seed, etc

@CloCkWeRX

CloCkWeRX Jun 23, 2014

Contributor

db:create I'm not super worried about, because it's idempotent.

db:seed in most projects I work on tends to be for setting up UI values, etc, and you'd want the "current" seeds always. Here we use it for 'first install' a bit, to demo things.

We should really consider something like FFCRM's setup task or similar to split the two roles (if we actually do any DB seeding for non demo purposes).

@0xdevalias

0xdevalias Jun 23, 2014

Member

I'd definitely be a fan of seperating any 'normal' db setup stuff from example data.

It's been ages since I looked at it, but I think that seed was only used for example stuff?

@CloCkWeRX

CloCkWeRX Jun 23, 2014

Contributor

Yeah that's what the rails tutorials encourage you to do.

With seeding/migrating/etc; I've ended up often shifting it into a separate task (ie: ci machine to cook things, then a separate build on ci machine to migrate dbs/seed data/etc) so you can configure various environments however you please; rather than re-inventing a bunch of toggling in a default attributes fashion.

Probably overkill for huginn at the moment, but maybe we should split the tasks up a bit, so you can call a provider to migrate/seed/etc however you please later on.

@0xdevalias

0xdevalias Jun 23, 2014

Member

+default['huginn']['user'] instead of hardcoded user?

+ end
+
+ before_restart do
+ rbenv_script "Huginn - Bundle, edit config, compile assets" do
@0xdevalias

0xdevalias Jun 23, 2014

Member

Looks like you do use the rbenv script stuff, just not for dev. Is that by choice, or an oversight?

@0xdevalias

0xdevalias Jun 23, 2014

Member

default['huginn']['user'] instead of hardcoded user?

+
+ notifies :enable, "service[nginx]"
+ notifies :start, "service[nginx]"
+end
@0xdevalias

0xdevalias Jun 23, 2014

Member

New line at end of file

+ })
+ code <<-EOH
+ sudo cp /home/huginn/shared/config/nginx.conf /etc/nginx/
+ rbenv sudo foreman export upstart /etc/init -a huginn -u huginn -l log
@0xdevalias

0xdevalias Jun 23, 2014

Member

Did you get this stuff working without error? I was having issues with weird stuff happening with unicorn threads not dying properly, and starting/stopping working really randomly/weirdly.

@CloCkWeRX

CloCkWeRX Jun 23, 2014

Contributor

It was a pain. I can't remember the precise thing that I did, and I remember having to kill processes a bit. I feel I did get it working for repeated deploys happily, but wouldn't bet my life on it.

@0xdevalias

0xdevalias Jun 23, 2014

Member

Haha fair enough. Will have a bit of a play and see how it goes :)

+
+
+case node['platform']
+when "centos"
@0xdevalias

0xdevalias Jun 23, 2014

Member

Does this work well?

I had:

+include_recipe 'apt' if platform?('ubuntu', 'debian')
 +include_recipe 'yum-epel' if platform_family?('rhel')
...
+if platform_family?('rhel')
@CloCkWeRX

CloCkWeRX Jun 23, 2014

Contributor

Yeah, I've let the underlying system work out which package manager to use and just assumed it is present. Given the small number of libraries with minor name differences, spitting out two lists seemed easiest.

@CloCkWeRX

CloCkWeRX Jun 23, 2014

Contributor

.. and yeah, it worked at least for the limited test cases I was doing. Happy to refactor to something better if there are more test platforms.

+ members ["huginn"]
+end
+
+bash "Setting huginn user with NOPASSWD option" do
@0xdevalias

0xdevalias Jun 23, 2014

Member

This ends up appending a new line to the huginn file everytime you run the chef command (mine)

+ shell "/bin/bash"
+end
+
+group "sudo"
@0xdevalias

0xdevalias Jun 23, 2014

Member

Do we want the group as a generic sudo, or should we stick to something like huginn (as was done before I think?)

Coverage Status

Coverage increased (+1.33%) when pulling d7e6651 on CloCkWeRX:refactor_deployment into e48938b on cantino:master.

Member

0xdevalias commented Jun 23, 2014

As per the wiki on a fresh digitalocean 1GB Ram 30GB SSD Disk San Francisco 1 CentOS 6.5 x64

gem install knife-solo
gem install librarian-chef
cd deployment && librarian-chef install
knife solo bootstrap root@huginn.example.com -r role[huginn_production]

Nginx error: We're sorry, but something went wrong.

[root@huginn huginn]# service --status-all
auditd (pid  804) is running...
crond (pid  964) is running...
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         

Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         

iscsi is stopped
iscsid is stopped
lvmetad is stopped
mdmonitor is stopped
multipathd is stopped
mysqld (pid  2233) is running...
netconsole module not loaded
Configured devices:
lo eth0
Currently active devices:
lo eth0
nginx (pid  2482) is running...
master (pid  954) is running...
rdisc is stopped
rsyslogd (pid  820) is running...
sandbox is stopped
saslauthd is stopped
openssh-daemon (pid  2626) is running...
svnserve is stopped
[root@huginn huginn]# service start huginn
start: unrecognized service

Edit: huginn start seems to be the right command.. though even though it says it starts/stops it, ps aux doesn't seem to show it running at all.

Doesn't look like the huginn service has been exported properly?

The files seem to be there though..

[root@huginn huginn]# ls /etc/init/*huginn*
/etc/init/huginn.conf  /etc/init/huginn-jobs-1.conf  /etc/init/huginn-jobs.conf  /etc/init/huginn-web-1.conf  /etc/init/huginn-web.conf
+ group "huginn"
+ rbenv_version node['rbenv']['ruby_version']
+ cwd "/home/huginn/current"
+ creates "/home/huginn/shared/RAKE-DB-CREATED"
@0xdevalias

0xdevalias Jun 23, 2014

Member

It seems you use the 'creates file' thing from my branch for the production stuff anyways.

Member

0xdevalias commented Jul 14, 2014

Found a bit of time to look at this again. (Running notes)

DigitalOcean, Fresh:1GB Ram 30GB SSD Disk San Francisco 1 CentOS 7.0 x64

gem install knife-solo
gem install librarian-chef

gem clean knife-solo
gem clean librarian-chef

cd deployment && librarian-chef install
knife solo bootstrap root@huginn.example.com -r role[huginn_production] -V -V | tee bootstrap-log.txt
Chef Client failed. 4 resources updated in 62.964061415 seconds
DEBUG: sudo -E -p 'knife sudo password: ' chef-solo -c ~/chef-solo/solo.rb -j ~/chef-solo/dna.json -l debug stdout: [2014-07-14T03:56:35-04:00] ERROR: mysql_service[default] (mysql::server line 20) had an error: ArgumentError: You must supply a name when declaring a package resource

[2014-07-14T03:56:35-04:00] ERROR: mysql_service[default] (mysql::server line 20) had an error: ArgumentError: You must supply a name when declaring a package resource
DEBUG: sudo -E -p 'knife sudo password: ' chef-solo -c ~/chef-solo/solo.rb -j ~/chef-solo/dna.json -l debug stdout: [2014-07-14T03:56:35-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

[2014-07-14T03:56:35-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
ERROR: RuntimeError: chef-solo failed. See output above.

https://github.com/opscode-cookbooks/mysql/blob/master/recipes/server.rb#L20

headdesk It seems everytime I throw time at this/my branch it ends in frustration, headaches, and loss of motivation.. :(

I'm super keen for this to actually be resolved properly, even to the point of throwing more $$ at the bounty if that's what it takes.

@cantino @dsander etc Do you guys know chef/know anyone that knows chef well that could get this resolved by chance?

Owner

cantino commented Jul 15, 2014

I don't know Chef very well. Maybe the version of Chef has changed?

I'm working on an Ansible deployment script which will hopefully work for people.

@cantino cantino referenced this pull request Jul 15, 2014

Closed

Deployment Strategies #401

Owner

cantino commented Jul 29, 2014

What state is our current Chef deployment in? Does it work?

Member

0xdevalias commented Jul 29, 2014

Havent checked since the other week, but didn't work for me then against latest CentOS. Didn't work for me earlier either.

I feel like it's probably 80% there from what I remember reviewing earlier. With a few more bits that could be pulled from my branch to improve it a little.

I was actually tempted the other day to start it all again from scratch, copy the bits as required/where they make sense, but start from an entirely clean slate.

Owner

cantino commented Jul 29, 2014

Okay, I'm going to make a new issue and unify this PR, they other PR, and the other issues into one.

@cantino cantino closed this Jul 29, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment