Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 123 lines (85 sloc) 4.31 kB
2242fb1 pulled in checklist and TODO from version_3
Philip (flip) Kromer authored
1 ### seamless stop-start support
2
3 * chef-client on bootup
4 - when you stop/start machines their IP address changes, so must reconverge
5
6 * create chef node for me
7
8 * chef needs to converge twice on hadoop master
9
10 * dirs are fucked up under natty beause paths are /dev/xvdi not /dev/sdi
11
12 ### cluster_service_discovery
13
14 * should let me concisely refer to another cluster for a service (or use the current server)
15 * Wait for service to announce
16
17 ### cluster_chef DSL
18
19 * `role` and `recipe`
20 - inject into the run_list directly
21 - `cluster_role_implication`
22 - clean up `first_boot.json`
23
24
25 ### Minor quibbles
26
27 * NFS server boostrapping
28 - need to upgrade kernel, restart
29
30 * A 'safety catch' -- see https://github.com/infochimps/cluster_chef/issues/18#issuecomment-1194916
31
32 * `use defaults`
33 * `ephemeral drives` cleanup
34
35 * Fog routines should use the cluster's region always -- https://github.com/infochimps/cluster_chef/issues/54
36
37 * ebs volumes shouldn't complain if data_bag missing
38
39 ### Concern Separation
40
41 Cluster chef currently consists of the following separate(able) concerns:
42
43 * **cluster_chef tools**
44 - the DSL that lets you define clusters
45 - the knife commands which use that DSL
46 - optional bootstrap scripts for a machine image that can then launch bootstrap-less
47 * **cluster-oriented cookbooks**
48 - `cluster_service_discovery` (recipes to let clusters discover services using chef-server's search)
49 - ?others?
50 * **cloud utility cookbooks**
51 - motd, system_internals (swappiness, ulimit, etc)
52 * **big data cookbooks** (hadoop, cassandra, redis, etc):
53 - cookbooks
54 - roles
55 - clusters
56
57 I think it's time to separate those into at least two repos.
58
59 REQUEST FOR COMMENTS:
60
61 #### Division of concerns
62
63 It's clear that the cluster_chef tools and the big data cookbooks should be divorced.
64
65 Proposed:
66
67 * `cluster_chef` holds only the DSL, knife commands, and bootstrap scripts -- basically the stuff in `lib/`, along with the gemspec etc.
68 * `cluster_chef-systems` -- holds cookbooks, roles and example clusters that use them.
69 - Utility cookbooks (`cluster_service_discovery`, motd, etc) and system cookbooks(hadoop, cassandra, etc) are housed in two separate folders.
70 - The standard layout would just include the cookbooks, but a cluster-oriented approach demands that the roles travel along too
71 * (possibly) `cluster_chef-chef-repo` (??better name, anyone??) -- a fork of https://github.com/opscode/chef-repo that integrates the above
72
73 #### Handling of cookbooks that originate from opscode-cookbooks
74
75 Right now we *copy* standard cookbooks from opscode's repo into the `cookbooks` directory. This lets us version them separately, but means we have to track them, and could cause conflicts with the majority of people who will be pulling from opscode-cookbooks already.
76
77 1. omit entirely, but list as dependencies (my vote)
78 2. `git subtree` pull them into the cluster_chef-cookbooks repo
79 3. copy them in as we've been doing
80 4. `git submodule` opscode-cookbooks and symlink
81
82 #### Organization of cookbooks repo
83
84 Opscode recommends a [standard layout for your chef repo](https://github.com/opscode/chef-repo). We should make the new arrangement work seamlessly within that structure.
85
86 The new layout should
87 * easy to integrate if you have your own existing chef-repo
88 * straightforward for a new chef user to adopt
89 * either mirror or be what we actually use
90
91 Revised proposal:
92
93 ```
94 clusters/
95 { actual clusters }
96 roles/
97 { roles }
98 { symlinks to things in vendor/cluster_chef/roles }
99
100 site-cookbooks/
101 { directories holding internal cookbooks }
102
103 cookbooks/
104 { symlinks into vendor/opscode-cookbooks }
105 { symlinks into vendor/cluster_chef-systems/site-cookbooks }
106
107 vendor/
108 opscode/cookbooks/ # git submodule of https://github.com/opscode/cookbooks
109 cluster_chef-systems/ # git submodule of https://github.com/infochimps/cluster_chef-systems
110 site-cookbooks/ # hadoop, cassandra, cluster_service_discovery, etc.
111 roles/
112 examples/
113 clusters/ # example clusters
114 roles/ # roles (if any) needed for just the example clusters
115
116 .chef/ # knife config, keypairs, etc
117 certificates/
118 config/
119 data_bags/
120 environments/
121 ```
122
Something went wrong with that request. Please try again.