Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 274 lines (242 sloc) 14.314 kB
f95325f Initial import.
Dell Openstack Crowbar Team authored
1 This file documents how to build the Crowbar installation DVD image.
2
3 Prerequisites:
4 * An unfiltered Internet connection and a decent amount of bandwidth.
5 The first time you try to build Crowbar it will need to download
9415a29 @VictorLowther Brought documentation for the build process up to date.
VictorLowther authored
6 all the packages that it will need to stage Crowbar onto an OS install
7 DVD.
f95325f Initial import.
Dell Openstack Crowbar Team authored
8 * bash version 4 or higher
9 build_crowbar.sh uses associative arrays, which bash got in version 4.
10 * mkisofs
11 The end product of the build script is an ISO image that will be used
12 to bootstrap the Crowbar admin node.
9415a29 @VictorLowther Brought documentation for the build process up to date.
VictorLowther authored
13 * debootstrap (if staging on to Ubuntu)
f95325f Initial import.
Dell Openstack Crowbar Team authored
14 The build process needs to download all the .debs and gems that
15 Crowbar requires, and we don't want to inadvertently mess up the build
16 machine when we do that. All extra packages are downloaded into a
17 chrooted minimal Ubuntu intall, and we use debootstrap to enable that.
18 * Sudo to root privileges for the following commands:
19 * /bin/mount, /bin/umount
20 We have to be able to mount and umount the Ubuntu .iso image, as well as
21 a tmpfs for debootstrap, and we have to be able to bind mount
22 /dev, /dev/pts, /proc, and /sys into the debootstrap chroot environment.
23 * /usr/sbin/debootstrap
24 debootstrap requires root privileges to run.
25 * /bin/cp
26 We need to copy things into and out of the debootstrap environment to
27 ensure it downloads and caches the right packages.
28 * /usr/sbin/chroot
29 All our package caching is done in a chroot environment, and chroot
30 requires root permissions to run.
31
32 If you want to allow build_crowbar to run the above commands as root
33 without having to enter a password each time, the build_crowbar.sh
34 includes a sample line you can fix up and add to /etc/sudoers.
35
9415a29 @VictorLowther Brought documentation for the build process up to date.
VictorLowther authored
36 Command Line Parameters:
37 * The first command line parameter is the OS you want to
38 stage Crowbar on to. Crowbar currently understands how to stage on to
1f4e496 @VictorLowther Add doc updates for the build process.
VictorLowther authored
39 ubuntu-10.10, redhat-5.6, redhat-5.7, and centos-5.7
9644c71 @VictorLowther Update build documentation.
VictorLowther authored
40
41 --update-cache
9415a29 @VictorLowther Brought documentation for the build process up to date.
VictorLowther authored
42 This parameter forces the build to try and update the build cache again.
9644c71 @VictorLowther Update build documentation.
VictorLowther authored
43 Use this parameter if you want to pull in updates from the upstream
44 repositories that Crowbar pulls packages from.
45
46 --merge|-m
9415a29 @VictorLowther Brought documentation for the build process up to date.
VictorLowther authored
47 Any arguments after this that do not start with a hyphen are interpreted as
48 branches in the git repository that you want to merge along with the current
49 branch into a throwaway branch before staging Crowbar on to the build .iso.
9644c71 @VictorLowther Update build documentation.
VictorLowther authored
50 If you are working on developing a barclamp or have changes in other
51 branches that you want to test out, this is the option for you. If your
52 build cache is also a git repository, it will try to merge identically
53 named branches in that repository as well -- this is a good way of testing
54 new packages before signing off on them. After the build is finished, the
55 build script will delete any throwaway branches it created, leaving the
56 repositories in the same state they started in.
57
58 --barclamps
59 Any arguments after this that do not start with a hyphen are interpreted
60 as extra barclamps to stage on the final .ISO. The build system will use
61 the metadata in the crowbar.yml files to figure out all the dependent
62 barclamps of the ones on the command line and include those as well. You
63 can also lass a barclamp group name my prefixing it with an @ sign, and
64 the build system will expand that into its component barclamps.
65
66 --test
67 This tells the build system to try and smoketest the freshly-generated
68 .ISO. Any arguments after this that do not begin with a hyphen are
69 passed to the test framework -- please see test_framework/README.testing
70 for more information.
71
72 --ci <barclamp> <branch>
73 This tells the build system to test <branch> of <barclamp> to see if it
74 merges cleanly in with the current codebase and passed the smoketest.
75 If it does, the build system will finalize the merge and update the
76 submodule reference in the main crowbar repository, otherwise it will
77 roll back the merges and fail. This option is intended to be used by
78 Jenkins to implement gated submodules for the Crowbar repository.
79
80 --shrink
81 This asks the build system to throw out unneeded packages from the base OS
82 install media. It relies on the presence of a minimal-install file listing
83 the minimal set of packages needed to deploy an admin node. Currently,
84 we only know how to do this on rpm based systems, because Ubuntu installs
85 are fairly small to begin with anyways.
9415a29 @VictorLowther Brought documentation for the build process up to date.
VictorLowther authored
86
1f4e496 @VictorLowther Add doc updates for the build process.
VictorLowther authored
87 --no-cache-update
88 This tells the build system that it should die instead of trying to update
89 the build cache. This is useful for automated build processes where you want
d480586 @VictorLowther Update build documentation
VictorLowther authored
90 to build Crowbar with a curated set of packages. When --no-cache-update is
91 present and the build cache is a Git repository, the current checkout of the
92 build cache will be recorded in the build-info file.
93
94 --no-iso
95 Do everything but actaully build the iso. This is useful for updating
96 any caches that need updating without actaully forcing a cache update.
1f4e496 @VictorLowther Add doc updates for the build process.
VictorLowther authored
97
98d0af2 @VictorLowther Epic dev rewrite to rationalize remote, release, and branch handling.
VictorLowther authored
98 --wild-cache
99 Create a temporary cache directory that is good for this build
100 only. The cache will be deleted when the build exits. This exists
101 mainly to test what a build that is pulling the latest stuff from
102 the Internet looks like. You should make sure that you are using
103 a caching web proxy such as polipo or squid, otherwise repeated
104 builds will be very slow.
105
f95325f Initial import.
Dell Openstack Crowbar Team authored
106 Usage:
9415a29 @VictorLowther Brought documentation for the build process up to date.
VictorLowther authored
107 * Run build_crowbar.sh from the Crowbar git repository.
f95325f Initial import.
Dell Openstack Crowbar Team authored
108 The first time it is run, it will download and cache all the files it
9415a29 @VictorLowther Brought documentation for the build process up to date.
VictorLowther authored
109 needs to stage Crowbar on to the OS installation DVD, build an OS install
110 ISO with Crowbar staged on to it, save the generated .ISO to the current
f95325f Initial import.
Dell Openstack Crowbar Team authored
111 working directory, and print out a message saying where it saved the image.
112 On subsequent runs it will run with the files it cached from the first
113 run, unless update-cache is passed as a parameter to the script.
114
115 Customization:
116
117 build_crowbar.sh has several different parameters you can tune, either from
118 $HOME/.build-crowbar.conf (for developer use), or from build-crowbar.conf
119 in the current directory (for automated builds).
120
121 Here are the parameters you can change through the above configuration files:
122 * DEBUG
123 If DEBUG is set to anything, build_crowbar will run in debug mode, and will
124 print a transcript of everything it is doing to standard error.
125 * CACHE_DIR
126 This is the default location where build_crowbar.sh will keep the files
127 it caches, along with the temporary directories used to mount the
128 ISO image, the debootstrap chroot, and the directory we perform the build
129 in. It defaults to $HOME/.crowbar-build-cache.
130 * ISO_LIBRARY
131 This is the default location where the Ubuntu .iso is stored. It defaults
132 to $CACHE_DIR/iso
133 * ISO_DEST
9415a29 @VictorLowther Brought documentation for the build process up to date.
VictorLowther authored
134 This is the location that we will save the Crowbar install image to.
f95325f Initial import.
Dell Openstack Crowbar Team authored
135 It defaults to the current directory.
136 * IMAGE_DIR
137 This is the location that we will mount isos in.
138 It defaults to $CACHE_DIR/image
139 * SLEDGEHAMMER_PXE_DIR
140 This points to the location we expect to find the unpacked Sledgehammer
141 PXE boot archive. It defaults to $CACHE_DIR/tftpboot
142 * VERSION
143 The default version of Crowbar. Defaults to dev.
c28a764 @galthaus DE225: Start renaming things.
galthaus authored
144 * BUILT_ISO
f95325f Initial import.
Dell Openstack Crowbar Team authored
145 The name of the ISO that build_crowbar.sh generates.
c28a764 @galthaus DE225: Start renaming things.
galthaus authored
146 Defaults to crowbar-$VERSION.iso
f95325f Initial import.
Dell Openstack Crowbar Team authored
147 * CROWBAR_DIR
148 The directory that the Crowbar source is cheched out to.
149 Defaults to the directory that build_crowbar.sh is in.
150 * VCS_CLEAN_CMD
151 This is the command that build_crowbar.sh will run to clean the tree before
152 staging the Crowbar build.
1f4e496 @VictorLowther Add doc updates for the build process.
VictorLowther authored
153 Defaults to 'git clean -f -d'
9415a29 @VictorLowther Brought documentation for the build process up to date.
VictorLowther authored
154
1688828 @VictorLowther Update README.build to include an overview what happens during a build.
VictorLowther authored
155 Build System Walkthrough:
156
157 When build_crowbar.sh is invoked, it performs the following processes in order:
158
159 1: Make sure we are in the C locale, and that $PATH is set to something sane.
160 2: Pick up any local configuration settings from $HOME/.build_crowbar.conf
161 or ./build_crowbar.conf
162 3: If USE_PROXY is set to 1, make sure that the HTTP and HTTPS proxy
163 settings are sane.
164 4: Set any uninitialized config variables to their defaults.
165 5: Source our generic build and test functions.
166 6: Figure out what OS we want to stage, and source the build and test
167 libraries. This will pull in the functions we need to actually stage
168 Crowbar on an OS install ISO. If we were asked to build on an OS that
169 we don't have build info for, die and print out the OSes we do know how
170 to stage things on.
171 7: Make sure that all the commands we will need to stage Crowbar on to an ISO
172 are installed on the system. If they are not, print a helpful error
173 message and die.
174 8: Grab the build lock to make sure that multiple builds do not stomp all
175 over eachother.
176 9: Do a little bookkeeping to make sure we are on a buildable Git branch.
177 If the build cache is in a git repository, record that information as well.
178 10: Parse our commandline options.
179 11: Make sure our essential build-related directories are present (including
180 the directory we will stage the build into, the directory we will mount
181 the ISO image on, and a chroot that will be used as part of the barclamp
182 staging process), and set up any build parameters that have not already
183 been set up.
184 12: If we were not passed a list of barclamps to install on the command line,
185 figure out what barclamps we need based on the submodule information from
186 the git branch we are on.
187 13: Pull in metadata from the crowbar.yml files for each barclamp.
188 This metadata will drive the rest of the install -- we need it to
189 figure out dependency relations between barclamps, what packages and files
190 to stage, and how to invoke any external build processes we might need.
45f36e1 @VictorLowther Update Sledgehammer documentation
VictorLowther authored
191 14: Make sure we have a Sledgehammer image handy, and build it if we don't.
1688828 @VictorLowther Update README.build to include an overview what happens during a build.
VictorLowther authored
192 15: If we don't have the OS ISO to stage on, and we know how to get one, then
193 download the .ISO we will need.
194 16: Clean out any leftovers from the last build, and make sure that
195 we don't inadvertently pull in any VCS cruft.
196 17: Mount the OS iso as a loopback file system, and index its package
197 pool if we don't already have it cached.
198 18: Stage some barclamp-independent build information into the build directory.
199 19: Create the build-info file for this build, and start adding useful
200 metadata into it.
201 20: Loop over the list of barclamps want to stage, and stage each one.
202 This is covered in more detail in the Barclamp Staging Walkthrough below.
203 21: Bundle each barclamp and its package cache into a per-barclamp tarball.
204 22: Create some legacy symlinks, and stage any custom proposals that this
205 iso will use.
206 23: Perform any OS specific fixups that are needed to make this image deploy
207 correctly.
208 24: Stage the Sledgehammer image.
209 25: Create the Crowbar .ISO by merging the contents of the build and the image
210 directory. Wherever there is a conflict in file names or contents, the
211 build directory has priority. If we were asked to generate a shrunken ISO,
212 that happens here.
213 26: If we were asked to test the ISO, invoke the test framework on our
214 newly-created ISO.
215 27: Clean up after ourselves.
216
217 Barclamp Staging Walkthrough:
218
219 build_crowbar will try to stage each barclamp in dependency order (as inferred
220 from each barclamp's crowbar.yml file). Staging a barclamp properly for the
221 OS we are staging on to requires the use of a chroot environment to ensure
222 that we get all the packages we need and that we don't break the host OS
223 in the process. Each barclamp is staged in 6 phases:
224
225 1: Check to see if all the OS packages listed in the pkgs: and build_pkgs:
226 section of the crowbar.yml are present in this barclamps's OS build cache.
227 If they are not, fetch any missing ones and all their dependencies using
228 a chroot environment, and add any new or updated packages so fetched back
229 into the build cache.
230 2: Check to see that all the gems listed in the crowbar.yml are present in
231 the build cache. If they are not, fetch them and all their dependencies in
232 the chroot environment, and add any new or updated gems to the build cache.
233 3: Download and cache any packages pointed to by raw_pkg stanzas in the
234 crowbar.yml that we are missing.
235 4: Download and cache any files required by extra_files stanzas in the
236 crowbar.yml thatwe are missing.
237 5: If the crowbar.yml has a build_cmd stanza, source that file and use it
238 to build an external package. The script pointed to by build_cmd should
239 have two functions declared:
240
241 bc_needs_build -- This function should return 0 if the external
242 pacakge needs building, and 1 if it does not.
243
244 bc_build: This function will be invoked after setting up a chroot and
245 bind-mounting the build cache for this barclamp into it. It is responsible
246 for using the chroot enviromnent to build the external package, and making
247 sure that the output of the build process winds up in the proper location
248 in the build cache so that the rest of the barclamp can properly use it.
249
250 bc-build has access to the following environment variables:
251 BC_DIR = the full path to the root of the barclamp source repository.
252 BC_CACHE = The full path to the barclamp package cache.
253
254 Any actual building should happen in the chroot environment.
255 To facilitate this, $BC_CACHE is bind-mounted to /mnt in the chroot,
256 any build_pkgs required by this barclamp will be installed in the chroot,
257 and /mnt/current_os in the chroot will be a symlink to the OS package cache
258 that for the barclamp build cache that is bind mounted to /mnt in the chroot.
259
260 You can use the chroot_install command to install any additional packages
261 you may need, and you can use the in_chroot command to run commands
262 in hte chroot environment.
263
264 For some in-tree examples, refer to the ganglia, provisioner, and deployer
265 barclamps. All of these have an additional script that is copied into the
266 chroot that handles most of the build tasks.
267
268 6: Tar up the source for the barclamp and the build cache into a deployable
269 barclamp tarball.
270
271
9415a29 @VictorLowther Brought documentation for the build process up to date.
VictorLowther authored
272 The build_crowbar.sh script is heavily commented, please refer to it for more
c28a764 @galthaus DE225: Start renaming things.
galthaus authored
273 detailed information.
Something went wrong with that request. Please try again.