Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse code

Import database version 0.99.1

  • Loading branch information...
commit 7e114b7293398d9e1a8981585160023d96840962 1 parent dd19f75
Seth Chisamore schisamo authored
36 cookbooks/database/README.md
Source Rendered
@@ -7,10 +7,17 @@ This cookbook is written primarily to use MySQL and the Opscode mysql cookbook.
7 7
8 8 This cookbook does not automatically restore database dumps, but does install tools to help with that.
9 9
  10 +Changes
  11 +=======
  12 +
  13 +## v0.99.1
  14 +
  15 +* Use Chef 0.10's `node.chef_environment` instead of `node['app_environment']`.
  16 +
10 17 Requirements
11 18 ============
12 19
13   -Chef 0.8 or higher required.
  20 +Chef 0.10.0 or higher required (for Chef environment use).
14 21
15 22 Platform
16 23 --------
@@ -29,8 +36,8 @@ The following Opscode cookbooks are dependencies:
29 36 Recipes
30 37 =======
31 38
32   -`ebs_volume`
33   -------------
  39 +ebs\_volume
  40 +-----------
34 41
35 42 Loads the aws information from the data bag. Searches the applications data bag for the database master or slave role and checks that role is applied to the node. Loads the EBS information and the master information from data bags. Uses the aws cookbook LWRP, `aws_ebs_volume` to manage the volume.
36 43
@@ -75,8 +82,8 @@ Deprecated Recipes
75 82
76 83 The following recipe is considered deprecated. It is kept for reference purposes.
77 84
78   -`ebs_backup`
79   -------------
  85 +ebs\_backup
  86 +-----------
80 87
81 88 Older style of doing mysql snapshot and replication using Adam Jacob's [ec2_mysql](http://github.com/adamhjk/ec2_mysql) script and library.
82 89
@@ -110,7 +117,7 @@ Usage
110 117
111 118 Aside from the application data bag (see the README in the application cookbook), create a role for the database master. Use a role.rb in your chef-repo, or create the role directly with knife.
112 119
113   - % knife role show my_app_database_master
  120 + % knife role show my_app_database_master -Fj
114 121 {
115 122 "name": "my_app_database_master",
116 123 "chef_type": "role",
@@ -126,23 +133,22 @@ Aside from the application data bag (see the README in the application cookbook)
126 133 }
127 134 }
128 135
129   -Also create a `production` role, or other role based on your desired environment. This is also used in the `application` cookbook.
  136 +Create a `production` environment. This is also used in the `application` cookbook.
130 137
131   - % knife role show production
  138 + % knife environment show production -Fj
132 139 {
133 140 "name": "production",
134   - "chef_type": "role",
135   - "json_class": "Chef::Role",
  141 + "description": "",
  142 + "cookbook_versions": {
  143 + },
  144 + "json_class": "Chef::Environment",
  145 + "chef_type": "environment",
136 146 "default_attributes": {
137   - "app_environment": "production"
138 147 },
139   - "description": "production environment role",
140   - "run_list": [
141   -
142   - ],
143 148 "override_attributes": {
144 149 }
145 150 }
  151 +
146 152
147 153 The cookbook `my_app_database` is recommended to set up any application specific database resources such as configuration templates, trending monitors, etc. It is not required, but you would need to create it separately in `site-cookbooks`. Add it to the `my_app_database_master` role.
148 154
4 cookbooks/database/metadata.json
... ... @@ -1,7 +1,7 @@
1 1 {
2 2 "name": "database",
3 3 "description": "Sets up the database master or slave",
4   - "long_description": "Database Cookbook\n=================\n\nConfigures mysql database masters and slaves and uses EBS for storage, integrating together with the application cookbook utilizing data bags for application related information.\n\nThis cookbook is written primarily to use MySQL and the Opscode mysql cookbook. Other RDBMS may be supported at a later date.\n\nThis cookbook does not automatically restore database dumps, but does install tools to help with that.\n\nRequirements\n============\n\nChef 0.8 or higher required.\n\nPlatform\n--------\n\nUbuntu, Debian, Red Hat, CentOS or Fedora.\n\nCookbooks\n---------\n\nThe following Opscode cookbooks are dependencies:\n\n* mysql\n* xfs\n* aws\n\nRecipes\n=======\n\n`ebs_volume`\n------------\n\nLoads the aws information from the data bag. Searches the applications data bag for the database master or slave role and checks that role is applied to the node. Loads the EBS information and the master information from data bags. Uses the aws cookbook LWRP, `aws_ebs_volume` to manage the volume.\n\nOn a master node:\n* if we have an ebs volume already as stored in a data bag, attach it.\n* if we don't have the ebs information then create a new one and attach it.\n* store the volume information in a data bag via a ruby block.\n\nOn a slave node:\n* use the master volume information to generate a snapshot.\n* create the new volume from the snapshot and attach it.\n\nAlso on a master node, generate some configuration for running a snapshot via `chef-solo` from cron.\n\nOn a new filesystem volume, create as XFS, then mount it in /mnt, and also bind-mount it to the mysql data directory (default /var/lib/mysql).\n\nmaster\n------\n\nThis recipe no longer loads AWS specific information, and the database position for replication is no longer stored in a databag because the client might not have permission to write to the databag item. This may be handled in a different way at a future date.\n\nSearches the apps databag for applications, and for each one it will check that the specified database master role is set in both the databag and applied to the node's run list. Then, retrieves the passwords for `root`, `repl` and `debian` users and saves them to the node attributes. If the passwords are not found in the databag, it prints a message that they'll be generated by the mysql cookbook.\n\nThen it adds the application databag database settings to a hash, to use later.\n\nIt sets up the template resource for `/etc/mysql/app_grants.sql`, passing in the database settings so privileges for application specific database users can be created.\n\nThen it will iterate over the databases and create them with the `mysql_database` LWRP, detecting their presence with the mysql rubygem.\n\nslave\n-----\n\n_TODO_: Retrieve the master status from a data bag, then start replication using a ruby block. The replication status needs to be handled in some other way for now since the master recipe above doesn't actually set it in the databag anymore.\n\nsnapshot\n--------\n\nRun via Chef Solo. Retrieves the db snapshot configuration from the specified JSON file. Uses the `mysql_database` LWRP to lock and unlock tables, and does a filesystem freeze and EBS snapshot.\n\nDeprecated Recipes\n==================\n\nThe following recipe is considered deprecated. It is kept for reference purposes.\n\n`ebs_backup`\n------------\n\nOlder style of doing mysql snapshot and replication using Adam Jacob's [ec2_mysql](http://github.com/adamhjk/ec2_mysql) script and library.\n\nData Bags\n=========\n\nThis cookbook uses the apps data bag item for the specified application; see the `application` cookbook's README.md. It also creates data bag items in a bag named 'aws' for storing volume information. In order to interact with EC2, it expects aws to have a main item:\n\n {\n \"id\": \"main\",\n \"ec2_private_key\": \"private key as a string\",\n \"ec2_cert\": \"certificate as a string\",\n \"aws_account_id\": \"\",\n \"aws_secret_access_key\": \"\",\n \"aws_access_key_id\": \"\"\n }\n\nNote: with the Open Source Chef Server, the server using the database recipes must be an admin client or it will not be able to create data bag items. You can modify whether the client is admin by editing it with knife.\n\n knife client edit <client_name>\n {\n ...\n \"admin\": true\n ...\n }\n\nThis is not required if the Chef Server is the Opscode Platform, instead use the ACL feature to modify access for the node to be able to update the data bag.\n\nUsage\n=====\n\nAside from the application data bag (see the README in the application cookbook), create a role for the database master. Use a role.rb in your chef-repo, or create the role directly with knife.\n\n % knife role show my_app_database_master\n {\n \"name\": \"my_app_database_master\",\n \"chef_type\": \"role\",\n \"json_class\": \"Chef::Role\",\n \"default_attributes\": {\n },\n \"description\": \"\",\n \"run_list\": [\n \"recipe[mysql::server]\",\n \"recipe[database::master]\"\n ],\n \"override_attributes\": {\n }\n }\n\nAlso create a `production` role, or other role based on your desired environment. This is also used in the `application` cookbook.\n\n % knife role show production\n {\n \"name\": \"production\",\n \"chef_type\": \"role\",\n \"json_class\": \"Chef::Role\",\n \"default_attributes\": {\n \"app_environment\": \"production\"\n },\n \"description\": \"production environment role\",\n \"run_list\": [\n\n ],\n \"override_attributes\": {\n }\n }\n\nThe cookbook `my_app_database` is recommended to set up any application specific database resources such as configuration templates, trending monitors, etc. It is not required, but you would need to create it separately in `site-cookbooks`. Add it to the `my_app_database_master` role.\n\nLicense and Author\n==================\n\nAuthor:: Adam Jacob (<adam@opscode.com>)\nAuthor:: Joshua Timberman (<joshua@opscode.com>)\nAuthor:: AJ Christensen (<aj@opscode.com>)\nAuthor:: Seth Chisamore (<schisamo@opscode.com>)\n\nCopyright 2009-2011, Opscode, Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n",
  4 + "long_description": "Database Cookbook\n=================\n\nConfigures mysql database masters and slaves and uses EBS for storage, integrating together with the application cookbook utilizing data bags for application related information.\n\nThis cookbook is written primarily to use MySQL and the Opscode mysql cookbook. Other RDBMS may be supported at a later date.\n\nThis cookbook does not automatically restore database dumps, but does install tools to help with that.\n\nChanges\n=======\n\n## v0.99.1\n\n* Use Chef 0.10's `node.chef_environment` instead of `node['app_environment']`.\n\nRequirements\n============\n\nChef 0.10.0 or higher required (for Chef environment use).\n\nPlatform\n--------\n\nUbuntu, Debian, Red Hat, CentOS or Fedora.\n\nCookbooks\n---------\n\nThe following Opscode cookbooks are dependencies:\n\n* mysql\n* xfs\n* aws\n\nRecipes\n=======\n\nebs\\_volume\n-----------\n\nLoads the aws information from the data bag. Searches the applications data bag for the database master or slave role and checks that role is applied to the node. Loads the EBS information and the master information from data bags. Uses the aws cookbook LWRP, `aws_ebs_volume` to manage the volume.\n\nOn a master node:\n* if we have an ebs volume already as stored in a data bag, attach it.\n* if we don't have the ebs information then create a new one and attach it.\n* store the volume information in a data bag via a ruby block.\n\nOn a slave node:\n* use the master volume information to generate a snapshot.\n* create the new volume from the snapshot and attach it.\n\nAlso on a master node, generate some configuration for running a snapshot via `chef-solo` from cron.\n\nOn a new filesystem volume, create as XFS, then mount it in /mnt, and also bind-mount it to the mysql data directory (default /var/lib/mysql).\n\nmaster\n------\n\nThis recipe no longer loads AWS specific information, and the database position for replication is no longer stored in a databag because the client might not have permission to write to the databag item. This may be handled in a different way at a future date.\n\nSearches the apps databag for applications, and for each one it will check that the specified database master role is set in both the databag and applied to the node's run list. Then, retrieves the passwords for `root`, `repl` and `debian` users and saves them to the node attributes. If the passwords are not found in the databag, it prints a message that they'll be generated by the mysql cookbook.\n\nThen it adds the application databag database settings to a hash, to use later.\n\nIt sets up the template resource for `/etc/mysql/app_grants.sql`, passing in the database settings so privileges for application specific database users can be created.\n\nThen it will iterate over the databases and create them with the `mysql_database` LWRP, detecting their presence with the mysql rubygem.\n\nslave\n-----\n\n_TODO_: Retrieve the master status from a data bag, then start replication using a ruby block. The replication status needs to be handled in some other way for now since the master recipe above doesn't actually set it in the databag anymore.\n\nsnapshot\n--------\n\nRun via Chef Solo. Retrieves the db snapshot configuration from the specified JSON file. Uses the `mysql_database` LWRP to lock and unlock tables, and does a filesystem freeze and EBS snapshot.\n\nDeprecated Recipes\n==================\n\nThe following recipe is considered deprecated. It is kept for reference purposes.\n\nebs\\_backup\n-----------\n\nOlder style of doing mysql snapshot and replication using Adam Jacob's [ec2_mysql](http://github.com/adamhjk/ec2_mysql) script and library.\n\nData Bags\n=========\n\nThis cookbook uses the apps data bag item for the specified application; see the `application` cookbook's README.md. It also creates data bag items in a bag named 'aws' for storing volume information. In order to interact with EC2, it expects aws to have a main item:\n\n {\n \"id\": \"main\",\n \"ec2_private_key\": \"private key as a string\",\n \"ec2_cert\": \"certificate as a string\",\n \"aws_account_id\": \"\",\n \"aws_secret_access_key\": \"\",\n \"aws_access_key_id\": \"\"\n }\n\nNote: with the Open Source Chef Server, the server using the database recipes must be an admin client or it will not be able to create data bag items. You can modify whether the client is admin by editing it with knife.\n\n knife client edit <client_name>\n {\n ...\n \"admin\": true\n ...\n }\n\nThis is not required if the Chef Server is the Opscode Platform, instead use the ACL feature to modify access for the node to be able to update the data bag.\n\nUsage\n=====\n\nAside from the application data bag (see the README in the application cookbook), create a role for the database master. Use a role.rb in your chef-repo, or create the role directly with knife.\n\n % knife role show my_app_database_master -Fj\n {\n \"name\": \"my_app_database_master\",\n \"chef_type\": \"role\",\n \"json_class\": \"Chef::Role\",\n \"default_attributes\": {\n },\n \"description\": \"\",\n \"run_list\": [\n \"recipe[mysql::server]\",\n \"recipe[database::master]\"\n ],\n \"override_attributes\": {\n }\n }\n\nCreate a `production` environment. This is also used in the `application` cookbook.\n\n % knife environment show production -Fj\n {\n \"name\": \"production\",\n \"description\": \"\",\n \"cookbook_versions\": {\n },\n \"json_class\": \"Chef::Environment\",\n \"chef_type\": \"environment\",\n \"default_attributes\": {\n },\n \"override_attributes\": {\n }\n }\n \n\nThe cookbook `my_app_database` is recommended to set up any application specific database resources such as configuration templates, trending monitors, etc. It is not required, but you would need to create it separately in `site-cookbooks`. Add it to the `my_app_database_master` role.\n\nLicense and Author\n==================\n\nAuthor:: Adam Jacob (<adam@opscode.com>)\nAuthor:: Joshua Timberman (<joshua@opscode.com>)\nAuthor:: AJ Christensen (<aj@opscode.com>)\nAuthor:: Seth Chisamore (<schisamo@opscode.com>)\n\nCopyright 2009-2011, Opscode, Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n",
5 5 "maintainer": "Opscode, Inc.",
6 6 "maintainer_email": "cookbooks@opscode.com",
7 7 "license": "Apache 2.0",
@@ -39,5 +39,5 @@
39 39 "database::master": "Creates application specific user and database",
40 40 "database::snapshot": "Locks tables and freezes XFS filesystem for replication, assumes EC2 + EBS"
41 41 },
42   - "version": "0.99.0"
  42 + "version": "0.99.1"
43 43 }
2  cookbooks/database/metadata.rb
@@ -3,7 +3,7 @@
3 3 license "Apache 2.0"
4 4 description "Sets up the database master or slave"
5 5 long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
6   -version "0.99.0"
  6 +version "0.99.1"
7 7
8 8 recipe "database", "Empty placeholder"
9 9 recipe "database::ebs_backup", "Considered deprecated, older way of backing up EBS volumes"
4 cookbooks/database/recipes/ebs_backup.rb
@@ -35,7 +35,7 @@
35 35 db_master_role = app["database_master_role"]
36 36 end
37 37
38   -ebs_info = Chef::DataBagItem.load(:aws, "ebs_#{db_master_role}_#{node['app_environment']}")
  38 +ebs_info = Chef::DataBagItem.load(:aws, "ebs_#{db_master_role}_#{node.chef_environment}")
39 39
40 40 gem_package "dbi"
41 41 gem_package "dbd-mysql"
@@ -78,7 +78,7 @@
78 78 end
79 79 end
80 80
81   -if db_type == "master" && node['app_environment'] == "production"
  81 +if db_type == "master" && node.chef_environment == "production"
82 82 template "/etc/cron.d/db-backup" do
83 83 source "ebs-backup-cron.erb"
84 84 owner "root"
22 cookbooks/database/recipes/ebs_volume.rb
@@ -44,8 +44,8 @@
44 44 if (app["database_master_role"] & node.run_list.roles).length == 1 || (app["database_slave_role"] & node.run_list.roles).length == 1
45 45 master_role = app["database_master_role"]
46 46 slave_role = app["database_slave_role"]
47   - root_pw = app["mysql_root_password"][node.app_environment]
48   - snapshots_to_keep = app["snapshots_to_keep"][node.app_environment]
  47 + root_pw = app["mysql_root_password"][node.chef_environment]
  48 + snapshots_to_keep = app["snapshots_to_keep"][node.chef_environment]
49 49
50 50 if (master_role & node.run_list.roles).length == 1
51 51 db_type = "master"
@@ -60,27 +60,27 @@
60 60 end
61 61
62 62 begin
63   - ebs_info = Chef::DataBagItem.load(:aws, "ebs_#{db_role}_#{node[:app_environment]}")
  63 + ebs_info = Chef::DataBagItem.load(:aws, "ebs_#{db_role}_#{node.chef_environment}")
64 64 Chef::Log.info("Loaded #{ebs_info['volume_id']} from DataBagItem aws[#{ebs_info['id']}]")
65 65 rescue
66   - Chef::Log.warn("Could not find the 'ebs_#{db_role}_#{node[:app_environment]}' item in the 'aws' data bag")
  66 + Chef::Log.warn("Could not find the 'ebs_#{db_role}_#{node.chef_environment}' item in the 'aws' data bag")
67 67 ebs_info = Hash.new
68 68 end
69 69
70 70 begin
71   - master_info = Chef::DataBagItem.load(:aws, "ebs_#{master_role}_#{node.app_environment}")
  71 + master_info = Chef::DataBagItem.load(:aws, "ebs_#{master_role}_#{node.chef_environment}")
72 72 Chef::Log.info "Loaded #{master_info['volume_id']} from DataBagItem aws[#{master_info['id']}]"
73 73 rescue
74 74 Chef::Application.fatal! "Could not load replication masters snapshot details", -41 if db_type == "slave"
75 75 end
76 76
77   - ruby_block "store_#{db_role}_#{node[:app_environment]}_volid" do
  77 + ruby_block "store_#{db_role}_#{node.chef_environment}_volid" do
78 78 block do
79   - ebs_vol_id = node[:aws][:ebs_volume]["#{db_role}_#{node[:app_environment]}"][:volume_id]
  79 + ebs_vol_id = node[:aws][:ebs_volume]["#{db_role}_#{node.chef_environment}"][:volume_id]
80 80
81 81 unless ebs_info['volume_id']
82 82 item = {
83   - "id" => "ebs_#{db_role}_#{node[:app_environment]}",
  83 + "id" => "ebs_#{db_role}_#{node.chef_environment}",
84 84 "volume_id" => ebs_vol_id
85 85 }
86 86 Chef::Log.info "Storing volume_id #{item.inspect}"
@@ -94,7 +94,7 @@
94 94 action :nothing
95 95 end
96 96
97   - aws_ebs_volume "#{db_role}_#{node[:app_environment]}" do
  97 + aws_ebs_volume "#{db_role}_#{node.chef_environment}" do
98 98 aws_access_key aws['aws_access_key_id']
99 99 aws_secret_access_key aws['aws_secret_access_key']
100 100 size 50
@@ -111,7 +111,7 @@
111 111 else
112 112 action [ :create, :attach ]
113 113 end
114   - notifies :create, resources(:ruby_block => "store_#{db_role}_#{node[:app_environment]}_volid")
  114 + notifies :create, resources(:ruby_block => "store_#{db_role}_#{node.chef_environment}_volid")
115 115 when "slave"
116 116 if master_info['volume_id']
117 117 snapshot_id master_info['volume_id']
@@ -141,7 +141,7 @@
141 141 'db_snapshot' => {
142 142 'ebs_vol_dev' => node.mysql.ec2_path,
143 143 'db_role' => db_role,
144   - 'app_environment' => node.app_environment,
  144 + 'app_environment' => node.chef_environment,
145 145 'username' => 'root',
146 146 'password' => root_pw,
147 147 'aws_access_key_id' => aws['aws_access_key_id'],
8 cookbooks/database/recipes/master.rb
@@ -31,12 +31,12 @@
31 31 (app['database_master_role'] & node.run_list.roles).each do |dbm_role|
32 32 %w{ root repl debian }.each do |user|
33 33 user_pw = app["mysql_#{user}_password"]
34   - if !user_pw.nil? and user_pw[node.app_environment]
  34 + if !user_pw.nil? and user_pw[node.chef_environment]
35 35 Chef::Log.debug("Saving password for #{user} as node attribute node['mysql']['server_#{user}_password'")
36   - node.set['mysql']["server_#{user}_password"] = user_pw[node.app_environment]
  36 + node.set['mysql']["server_#{user}_password"] = user_pw[node.chef_environment]
37 37 node.save
38 38 else
39   - log "A password for MySQL user #{user} was not found in DataBag 'apps' item '#{app["id"]}' for environment ' for #{node.app_environment}'." do
  39 + log "A password for MySQL user #{user} was not found in DataBag 'apps' item '#{app["id"]}' for environment ' for #{node.chef_environment}'." do
40 40 level :warn
41 41 end
42 42 log "A random password will be generated by the mysql cookbook and added as 'node.mysql.server_#{user}_password'. Edit the DataBag item to ensure it is set correctly on new nodes" do
@@ -79,7 +79,7 @@
79 79 search(:apps) do |app|
80 80 (app['database_master_role'] & node.run_list.roles).each do |dbm_role|
81 81 app['databases'].each do |env,db|
82   - if env =~ /#{node['app_environment']}/
  82 + if env =~ /#{node.chef_environment}/
83 83 root_pw = node['mysql']['server_root_password']
84 84 mysql_database "create #{db['database']}" do
85 85 host "localhost"

0 comments on commit 7e114b7

Please sign in to comment.
Something went wrong with that request. Please try again.