Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 197 lines (150 sloc) 10.994 kb
ae1e5cb Nathaniel Eliot From conversation with Flip: the remainder of #81 should be a TODO item,...
temujin9 authored
1 * refactor standard dirs and volume_dirs into a single, more coherent resource
2
f15120d Philip (flip) Kromer permanent attribute works
mrflip authored
3 * We should use the `Fog::Compute::AWS::FLAVORS` constant that [fog defines](http://rubydoc.info/github/fog/fog/master/Fog/Compute/AWS) in the cloud code (instead of the one we put there)
66dadc9 Philip (flip) Kromer TODO: a `smush` method to replace the ugly-but-necessary Mash.new().merg...
mrflip authored
4
5 * All over the place there is the following construct (absolutely necessary, absolutely horrid):
6
7 foo = Mash.new().merge(node[:system]).merge(node[:system][:component])
8
9 You might look at this and think "gee I know a much simpler way to do that". That simpler way does not work; this way does.
10
11 I propose adding a 'smush' method to `silverware/libraries/cookbook_utils`:
12
13 ```ruby
14 module Ironfan::CookbookUtils
15 module_function
16
17 # Merge the given objects (node attributes, hashes, or anything
18 # else with `#to_hash`) into a combined `Mash` object. Objects
19 # given later in the list 'win' over objects given earlier.
20 #
21 # @examples
22 # template_vars = Ironfan::CookbookUtils.smush( node[:flume], node[:flume][:agent], :zookeeper_port => node[:zookeeper][:port] )
23 #
24 # @param [Array[#to_hash]] smushables -- any number of things that respond to `#to_hash`
25 #
26 def smush(*smushables)
27 result = Mash.new
28 smushables.compact.each do |smushable|
29 result.merge! smushable.to_hash
30 end
31 result
32 end
33
34 end
35
36 (obviously the hard part is not writing the method, it's applying it to all the cookbooks.)
37
38
39
d3ea9e8 Philip (flip) Kromer attribute normalization
mrflip authored
40 ### Knife commands
41
6e3ba80 Philip (flip) Kromer got el_ridiculoso-grande working: runs like 30 components simultaneously...
mrflip authored
42 * knife cluster launch should fail differently if you give it a facet that doesn't exist
3a22d4b Philip (flip) Kromer a couple TODOs from the issues
mrflip authored
43
44
b539bd8 Philip (flip) Kromer holy crap, knife cluster vm [CMD] [SLICE] works now. It's rough, but pre...
mrflip authored
45 * reify notion of 'homebase'; cluster commands work off it
46 * move away from referring to Chef::Config everywhere;
47
48
49 __________________________________________________________________________
50
51 Rename completed,
52
53 New homes:
54
55 * **http://github.com/infochimps-labs/ironfan** -- the primary destination, and the home of the knife tools (pretty much what's here in `infochimps/cluster_chef`)
56 * **http://github.com/infochimps-labs/ironfan-pantry** -- collection of public cookbooks, roles and demo clusters
57 * **http://github.com/infochimps-labs/ironfan-homebase** -- skeleton homebase (looks a lot like the current cluster_chef-homebase)
58 * **http://github.com/infochimps-labs/ironfan-ci** -- continuous integration stuff (homebase/vagrants)
59 * **http://github.com/infochimps-labs/ironfan-scrubby** -- de-linter/simulator (`cluster_chef/lib/cluster_chef/cookbook_munger*`)
60 * **http://github.com/infochimps-labs/opscode-cookbooks** -- our fork of the opscode cookbooks repo
61
62 ## Phase 1: Regex Replace
63
64 0. DONE warn about the name change. Since the `version_3` branch is left pre-namechange, decision was made to do a pull request when things are pull-able, not spam people before.
65 1. DONE Make a branch in each repo (`cluster_chef` and `*-homebase`) called `before_rename`, holding the current state of the code. Make another branch `ironfan`, where the renames will occur. That branch will be deleted as soon as the merge lands, hopefully before anyone even notices.
66 2. DONE Make sure `el_ridiculoso` is up to date and that it fully converges in the vagrant and cloud environments.
67 3. DONE Ensure all infochimps devs have push-pulled to various repos. make a tarball of the repos (.git and everything) and put them somewhere safe.
68 4. DONE rename the `vendor/infochimps/metachef` cookbook as vendor/infochimps/silverware.
69 5. DONE Bump the *minor* version number on all cookbooks (so 3.0.x => 3.1.x). Commit.
70 6. DONE regex-replace `ClusterChef` => `Ironfan`, `cluster_chef` to `ironfan` and `[Mm]etachef` => `[Ss]ilverware`. Do this in `cluster_chef` and `infochimps-labs/ironfan-homebase` only. Get chef-client to complete on el_ridiculoso using an emptied-and-reloaded local chef server. Run knife cluster sync on all clusters in all homebases against local chef server.
71 7. ...
72 12. DONE once the name is updated in the gemspec, release the `ironfan` gem. @temujin9 and I agree that it should be a single combined gem now that we won't have the stupid `_chef` gem name conflict.
3a22d4b Philip (flip) Kromer a couple TODOs from the issues
mrflip authored
73
b539bd8 Philip (flip) Kromer holy crap, knife cluster vm [CMD] [SLICE] works now. It's rough, but pre...
mrflip authored
74 ## Phase 2: Repo migration
3a22d4b Philip (flip) Kromer a couple TODOs from the issues
mrflip authored
75
b539bd8 Philip (flip) Kromer holy crap, knife cluster vm [CMD] [SLICE] works now. It's rough, but pre...
mrflip authored
76 1. DONE back up the repo and put it somewhere safe.
77 2. DONE Transfer ownership of `infochimps/cluster_chef` to `infochimps-labs/cluster_chef`. Do whatever is necessary/possible to migrate issues over.
78 3. DONE Change name of `infochimps-labs/cluster_chef` to `infochimps-labs/ironfan`. Created new repo `infochimps-labs/ironfan-homebase`. 4. DONE Create new repo `infochimps/cluster_chef` with helpful link to new repo. I don't think we need a placeholder at `infochimps-labs/cluster_chef-homebase`.
79 5. DONE Rename master branch of old cluster chef to be `version_2`, and freeze the `version_3`. Both now carry a deprecation warning pointing you to ironfan.
80 6. IN PROGRESS Update any remaining documentation links, etc to point to new home.
81 7. IN PROGRESS Have all devs edit their .git/config to point at right place.
3a22d4b Philip (flip) Kromer a couple TODOs from the issues
mrflip authored
82
b539bd8 Philip (flip) Kromer holy crap, knife cluster vm [CMD] [SLICE] works now. It's rough, but pre...
mrflip authored
83 so here is the dev-level worksforme -- @temujin9 will close once he's given it the sysadmin-level worksforme
2974ad0 Nathaniel Eliot Dumping old issues triage into TODO
temujin9 authored
84
85
86
87
88
89
90 # Old Issues Triage
91 From https://github.com/infochimps-labs/ironfan/issues/102 with love. Deleted ownership, and everything that was done or has its own issue.
92
93 ## Must Do
94 * merge volumes into silverware. merge ebs_volumes into ec2 cookbook
95 * Basic CI testing of cookbooks
96 * RSpecs for silverback (lib and knife tools)
97 * RSpecs for silverware are mostly in place -- ensure they are.
98 * push cookbooks to community.opscode.com
99 * refine and explain updated git workflow
100
101 ## Docco
102 Use the [opscode EC2 fast start](http://wiki.opscode.com/display/chef/EC2+Bootstrap+Fast+Start+Guide) as a guide -- our getting started should start at the same place, and cover the same detail as the EC2 bootstrap guide.
103
104 * Clear description of metadiscovery
105 * make sure README files in cookbooks aren’t wildly inaccurate
106 * Carry out setup directions, ensure they work:
107 - cluster_chef if you’re using our homebase
108 - cluster_chef if you’re using opscode’s homebase
109 * local vagrant environment
110 * hadoop cluster bootstrapping
111
112 ## Piddly Shit
113
114 * standardize the `zabbix` cookbook (no more /opt, etc -- more in the TODO)
115 * volumes don't deep merge -- eg you have to mount_ephemerals in the facet if you modify htem
116 * kill_old_service should disable services (may be leaving /etc/rc.d cruft).
117 * kill old service doesn't go the first time. why?
118 * chef client/server cookbook: set chef user UID / GID; client can set log directory
119 * apt has a dashboard at http://{hostname}:3142/report
120 * can use knife ssh as me@ or as ubuntu@
121 * knife command to set/remove permanent on a node + disableApiTermination on box. knife cluster kill refuses to delete nodes with permanent set. knife cluster sync sets permanent on if permanent(true), removes if permanent(false), ignores if permanent nil or unset.
122 * style-guide alignment (prefix_root becomes prefix)
123
124 ## Really Want
125
126 * unify the hashlike underpinning to be same across silverware & cluster_chef. Make sure we love (or accept) all the differences between it and Gorrillib’s, and between it and Chef’s.
127 * Keys are transmitted in databags, using a helper, and not in node attributes
128 * easy to create a dummy node (load balancer, external resource, etc)
129 * components can have arbitrary attributes (kinda. they take an `:info` param, behavior which may change later)
130 * All cookbooks have nice detailed announcements
131 * full roll out of log_integration, monitoring
132 * Git deploy abstraction similar to `install_from`
133
134 ## Cookbook checklist:
135 * Validate all the cookbooks against checklist -- see notes/README-checklist.md
136
137 | flip fixed | temujin9 checked |
138 +------------+------------------+
139 cassandra | | |
140 ec2 | | |
141 elasticsearch | | |
142 firewall | | |
143 flume | | |
144 ganglia | | |
145 graphite | | |
146 hadoop_cluster | | |
147 hbase | | |
148 hive | | |
149 jenkins | | |
150 jruby | | |
151 nfs | | |
152 nodejs | | |
153 papertrail | | |
154 pig | | |
155 redis | | |
156 resque | | |
157 Rstats | | |
158 statsd | | |
159 zookeeper | | |
160 # meta:
161 install_from | | |
162 motd | | |
163 mountable_volumes | | |
164 provides_service | | |
165 # Need thinkin':
166 big_package | | |
167 cluster_chef | | |
168
169
170 ## Things that are probably straightforward to fix as soon as we know how
171
172 * announcements should probably be published very early, but they need to know lots about the machine YUK
173 * split between clusters / roles / integration cookbooks
174 * inheritance of clusters
175
176 ## Things We Hate But Might Have to Continue Hating
177
178 * Cluster refactor -- clusters / stacks / components, not clusters / roles / cookbooks
179 * move cluster discovery to cloud class.
180 * Server#normalize! doesn’t imprint object (ie. server attributes poke through to the facet & cluster, rather than being *set* on the object)
181 * The fact you can only see one cluster at a time is stupid.
182 * security group pairing is sucky.
183 * ubuntu home drive bullshit
184 * Finer-grained security group control (eg nfs server only opens a couple ports, not all)
185 * nfs recipe uses discovery right (thus allowing more than one NFS share to exist in the universe)
186 * roles UGGGHHHHAERWSDFKHSBLAH
187
188 ## Ponies!
189 * sync cookbooks up/down to `infochimps-cookbooks/` 
190   - note: infochimps-cookbooks the org will be dereferenced in favor of ironfan-lib the single repo; it's unclear which pull requesters will prefer. We will do at least one push so that names and URLs are current, and we're not removing anything, but infochimps-cookbooks has an unclear future.
191 * foodcritic compatibility
192 * build out cookbook munger, make it less spike-y
193 * spot pricing
194 * rackspace compatibility
195 * cookbook munger reads comments in attributes file to populate metadata.rb
196 * `gem install ironfan; ironfan install` checks everything out
Something went wrong with that request. Please try again.