Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 195 lines (148 sloc) 10.727 kb
66dadc9 TODO: a `smush` method to replace the ugly-but-necessary Mash.new().m…
Philip (flip) Kromer authored
1
2
3 * All over the place there is the following construct (absolutely necessary, absolutely horrid):
4
5 foo = Mash.new().merge(node[:system]).merge(node[:system][:component])
6
7 You might look at this and think "gee I know a much simpler way to do that". That simpler way does not work; this way does.
8
9 I propose adding a 'smush' method to `silverware/libraries/cookbook_utils`:
10
11 ```ruby
12 module Ironfan::CookbookUtils
13 module_function
14
15 # Merge the given objects (node attributes, hashes, or anything
16 # else with `#to_hash`) into a combined `Mash` object. Objects
17 # given later in the list 'win' over objects given earlier.
18 #
19 # @examples
20 # template_vars = Ironfan::CookbookUtils.smush( node[:flume], node[:flume][:agent], :zookeeper_port => node[:zookeeper][:port] )
21 #
22 # @param [Array[#to_hash]] smushables -- any number of things that respond to `#to_hash`
23 #
24 def smush(*smushables)
25 result = Mash.new
26 smushables.compact.each do |smushable|
27 result.merge! smushable.to_hash
28 end
29 result
30 end
31
32 end
33
34 (obviously the hard part is not writing the method, it's applying it to all the cookbooks.)
35
36
37
d3ea9e8 attribute normalization
Philip (flip) Kromer authored
38 ### Knife commands
39
6e3ba80 got el_ridiculoso-grande working: runs like 30 components simultaneou…
Philip (flip) Kromer authored
40 * knife cluster launch should fail differently if you give it a facet that doesn't exist
3a22d4b a couple TODOs from the issues
Philip (flip) Kromer authored
41
42
b539bd8 holy crap, knife cluster vm [CMD] [SLICE] works now. It's rough, but …
Philip (flip) Kromer authored
43 * reify notion of 'homebase'; cluster commands work off it
44 * move away from referring to Chef::Config everywhere;
45
46
47 __________________________________________________________________________
48
49 Rename completed,
50
51 New homes:
52
53 * **http://github.com/infochimps-labs/ironfan** -- the primary destination, and the home of the knife tools (pretty much what's here in `infochimps/cluster_chef`)
54 * **http://github.com/infochimps-labs/ironfan-pantry** -- collection of public cookbooks, roles and demo clusters
55 * **http://github.com/infochimps-labs/ironfan-homebase** -- skeleton homebase (looks a lot like the current cluster_chef-homebase)
56 * **http://github.com/infochimps-labs/ironfan-ci** -- continuous integration stuff (homebase/vagrants)
57 * **http://github.com/infochimps-labs/ironfan-scrubby** -- de-linter/simulator (`cluster_chef/lib/cluster_chef/cookbook_munger*`)
58 * **http://github.com/infochimps-labs/opscode-cookbooks** -- our fork of the opscode cookbooks repo
59
60 ## Phase 1: Regex Replace
61
62 0. DONE warn about the name change. Since the `version_3` branch is left pre-namechange, decision was made to do a pull request when things are pull-able, not spam people before.
63 1. DONE Make a branch in each repo (`cluster_chef` and `*-homebase`) called `before_rename`, holding the current state of the code. Make another branch `ironfan`, where the renames will occur. That branch will be deleted as soon as the merge lands, hopefully before anyone even notices.
64 2. DONE Make sure `el_ridiculoso` is up to date and that it fully converges in the vagrant and cloud environments.
65 3. DONE Ensure all infochimps devs have push-pulled to various repos. make a tarball of the repos (.git and everything) and put them somewhere safe.
66 4. DONE rename the `vendor/infochimps/metachef` cookbook as vendor/infochimps/silverware.
67 5. DONE Bump the *minor* version number on all cookbooks (so 3.0.x => 3.1.x). Commit.
68 6. DONE regex-replace `ClusterChef` => `Ironfan`, `cluster_chef` to `ironfan` and `[Mm]etachef` => `[Ss]ilverware`. Do this in `cluster_chef` and `infochimps-labs/ironfan-homebase` only. Get chef-client to complete on el_ridiculoso using an emptied-and-reloaded local chef server. Run knife cluster sync on all clusters in all homebases against local chef server.
69 7. ...
70 12. DONE once the name is updated in the gemspec, release the `ironfan` gem. @temujin9 and I agree that it should be a single combined gem now that we won't have the stupid `_chef` gem name conflict.
3a22d4b a couple TODOs from the issues
Philip (flip) Kromer authored
71
b539bd8 holy crap, knife cluster vm [CMD] [SLICE] works now. It's rough, but …
Philip (flip) Kromer authored
72 ## Phase 2: Repo migration
3a22d4b a couple TODOs from the issues
Philip (flip) Kromer authored
73
b539bd8 holy crap, knife cluster vm [CMD] [SLICE] works now. It's rough, but …
Philip (flip) Kromer authored
74 1. DONE back up the repo and put it somewhere safe.
75 2. DONE Transfer ownership of `infochimps/cluster_chef` to `infochimps-labs/cluster_chef`. Do whatever is necessary/possible to migrate issues over.
76 3. DONE Change name of `infochimps-labs/cluster_chef` to `infochimps-labs/ironfan`. Created new repo `infochimps-labs/ironfan-homebase`. 4. DONE Create new repo `infochimps/cluster_chef` with helpful link to new repo. I don't think we need a placeholder at `infochimps-labs/cluster_chef-homebase`.
77 5. DONE Rename master branch of old cluster chef to be `version_2`, and freeze the `version_3`. Both now carry a deprecation warning pointing you to ironfan.
78 6. IN PROGRESS Update any remaining documentation links, etc to point to new home.
79 7. IN PROGRESS Have all devs edit their .git/config to point at right place.
3a22d4b a couple TODOs from the issues
Philip (flip) Kromer authored
80
b539bd8 holy crap, knife cluster vm [CMD] [SLICE] works now. It's rough, but …
Philip (flip) Kromer authored
81 so here is the dev-level worksforme -- @temujin9 will close once he's given it the sysadmin-level worksforme
2974ad0 @temujin9 Dumping old issues triage into TODO
temujin9 authored
82
83
84
85
86
87
88 # Old Issues Triage
89 From https://github.com/infochimps-labs/ironfan/issues/102 with love. Deleted ownership, and everything that was done or has its own issue.
90
91 ## Must Do
92 * merge volumes into silverware. merge ebs_volumes into ec2 cookbook
93 * Basic CI testing of cookbooks
94 * RSpecs for silverback (lib and knife tools)
95 * RSpecs for silverware are mostly in place -- ensure they are.
96 * push cookbooks to community.opscode.com
97 * refine and explain updated git workflow
98
99 ## Docco
100 Use the [opscode EC2 fast start](http://wiki.opscode.com/display/chef/EC2+Bootstrap+Fast+Start+Guide) as a guide -- our getting started should start at the same place, and cover the same detail as the EC2 bootstrap guide.
101
102 * Clear description of metadiscovery
103 * make sure README files in cookbooks aren’t wildly inaccurate
104 * Carry out setup directions, ensure they work:
105 - cluster_chef if you’re using our homebase
106 - cluster_chef if you’re using opscode’s homebase
107 * local vagrant environment
108 * hadoop cluster bootstrapping
109
110 ## Piddly Shit
111
112 * standardize the `zabbix` cookbook (no more /opt, etc -- more in the TODO)
113 * volumes don't deep merge -- eg you have to mount_ephemerals in the facet if you modify htem
114 * kill_old_service should disable services (may be leaving /etc/rc.d cruft).
115 * kill old service doesn't go the first time. why?
116 * chef client/server cookbook: set chef user UID / GID; client can set log directory
117 * apt has a dashboard at http://{hostname}:3142/report
118 * can use knife ssh as me@ or as ubuntu@
119 * knife command to set/remove permanent on a node + disableApiTermination on box. knife cluster kill refuses to delete nodes with permanent set. knife cluster sync sets permanent on if permanent(true), removes if permanent(false), ignores if permanent nil or unset.
120 * style-guide alignment (prefix_root becomes prefix)
121
122 ## Really Want
123
124 * unify the hashlike underpinning to be same across silverware & cluster_chef. Make sure we love (or accept) all the differences between it and Gorrillib’s, and between it and Chef’s.
125 * Keys are transmitted in databags, using a helper, and not in node attributes
126 * easy to create a dummy node (load balancer, external resource, etc)
127 * components can have arbitrary attributes (kinda. they take an `:info` param, behavior which may change later)
128 * All cookbooks have nice detailed announcements
129 * full roll out of log_integration, monitoring
130 * Git deploy abstraction similar to `install_from`
131
132 ## Cookbook checklist:
133 * Validate all the cookbooks against checklist -- see notes/README-checklist.md
134
135 | flip fixed | temujin9 checked |
136 +------------+------------------+
137 cassandra | | |
138 ec2 | | |
139 elasticsearch | | |
140 firewall | | |
141 flume | | |
142 ganglia | | |
143 graphite | | |
144 hadoop_cluster | | |
145 hbase | | |
146 hive | | |
147 jenkins | | |
148 jruby | | |
149 nfs | | |
150 nodejs | | |
151 papertrail | | |
152 pig | | |
153 redis | | |
154 resque | | |
155 Rstats | | |
156 statsd | | |
157 zookeeper | | |
158 # meta:
159 install_from | | |
160 motd | | |
161 mountable_volumes | | |
162 provides_service | | |
163 # Need thinkin':
164 big_package | | |
165 cluster_chef | | |
166
167
168 ## Things that are probably straightforward to fix as soon as we know how
169
170 * announcements should probably be published very early, but they need to know lots about the machine YUK
171 * split between clusters / roles / integration cookbooks
172 * inheritance of clusters
173
174 ## Things We Hate But Might Have to Continue Hating
175
176 * Cluster refactor -- clusters / stacks / components, not clusters / roles / cookbooks
177 * move cluster discovery to cloud class.
178 * Server#normalize! doesn’t imprint object (ie. server attributes poke through to the facet & cluster, rather than being *set* on the object)
179 * The fact you can only see one cluster at a time is stupid.
180 * security group pairing is sucky.
181 * ubuntu home drive bullshit
182 * Finer-grained security group control (eg nfs server only opens a couple ports, not all)
183 * nfs recipe uses discovery right (thus allowing more than one NFS share to exist in the universe)
184 * roles UGGGHHHHAERWSDFKHSBLAH
185
186 ## Ponies!
187 * sync cookbooks up/down to `infochimps-cookbooks/` 
188   - note: infochimps-cookbooks the org will be dereferenced in favor of ironfan-lib the single repo; it's unclear which pull requesters will prefer. We will do at least one push so that names and URLs are current, and we're not removing anything, but infochimps-cookbooks has an unclear future.
189 * foodcritic compatibility
190 * build out cookbook munger, make it less spike-y
191 * spot pricing
192 * rackspace compatibility
193 * cookbook munger reads comments in attributes file to populate metadata.rb
194 * `gem install ironfan; ironfan install` checks everything out
Something went wrong with that request. Please try again.