Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 286 lines (236 sloc) 12.979 kb
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
1 =================
2 Buildout Tutorial
3 =================
4
5 `Buildout <http://www.buildout.org/>`_ can seem far more complex than
6 it really is if you're new to it and are trying to understand an
7 `existing buildout
8 <https://github.com/plone/Installers-UnifiedInstaller/blob/master/buildout_templates/base.cfg>`_.
9 A single buildout can grow to do quite a lot, but a buildout also need
10 only be complex as what it's being used to do. This document aims to
008f160 @rpatterson Clarify the 'buildout' term using case to distinguish, from Spanky's
rpatterson authored
11 help the reader understand the core concepts of Buildout starting with
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
12 the simplest possible buildout and working through increasingly
13 complex buildouts. At the end, the reader should be able to walk up
14 to an existing complex buildout and understand what it does.
15
5720d4e @rpatterson Doesn't work on github anyways, so back to the sphinx way
rpatterson authored
16 .. toctree::
3abf98e @rpatterson Add a toc
rpatterson authored
17
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
18
19 Why Buildout?
20 =============
21
008f160 @rpatterson Clarify the 'buildout' term using case to distinguish, from Spanky's
rpatterson authored
22 The primary purpose of Buildout is reproducible deployments. A
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
23 deployment can be a number of things from as simple as installing a
24 python script in an isolated directory and environment to a full web
25 application deployment containing multiple pieces and a lot of
008f160 @rpatterson Clarify the 'buildout' term using case to distinguish, from Spanky's
rpatterson authored
26 configuration. To that end Buildout provides a number of services:
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
27
28 Describing Deployments
29 Buildout is best used to define and describe what a given deployment
30 needs to contain and how to configure it. The buildout can then be
31 used to reliably reproduce that deployment in multiple locations and
32 environments. You can keep your buildout in version control, check it
33 out in multiple locations, such as staging, production, and on each
34 team members workstation and trust that there is consistency between
35 them all.
36
37 Factoring Deployment Configuration
a9004e2 @rpatterson Ommitted a word
rpatterson authored
38 Another element of reproducible deployments is to allow sharing
39 common pieces of slightly different deployments. For example,
40 when building a web application you often want to use tools for
41 development that should never be deployed to production, but you
42 also need to be able to trust that your development environment
43 otherwise reflects what will be deployed to production. Buildout
44 provides ways to factor deployment configuration such that you can
45 use the same buildout for development and production but specify
46 which variation of the buildout should be used in each
47 environment.
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
48
49 Deployment Isolation
303a427 @rpatterson Add a link to more details on buildout python isolation
rpatterson authored
50 Another important element of reproducible deployments is
51 isolation, ensuring that the deployment won't be interfered with
52 or interfere with other things in the same environment. To that
53 end, a buildout deployment `provides an isolated Python
54 environment
55 <http://pypi.python.org/pypi/zc.buildout/1.5.2#system-python-and-zc-buildout-1-5>`_
008f160 @rpatterson Clarify the 'buildout' term using case to distinguish, from Spanky's
rpatterson authored
56 into which Buildout can install Python distributions which won't
303a427 @rpatterson Add a link to more details on buildout python isolation
rpatterson authored
57 interfere with anything outside the buildout or be interfered with
58 by things outside the buildout.
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
59
60 Installing Software
61 Finally, a big part of any real-world deployment is retrieving,
62 building and installing software. Another important part of that
63 is doing the same for any dependencies of that software. As such,
008f160 @rpatterson Clarify the 'buildout' term using case to distinguish, from Spanky's
rpatterson authored
64 Buildout uses `distribute
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
65 <http://packages.python.org/distribute/>`_ and provides it's own
66 version of `easy_install
6716917 @rpatterson typo
rpatterson authored
67 <http://packages.python.org/distribute/easy_install.html>`_ to do
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
68 this for Python software.
69
008f160 @rpatterson Clarify the 'buildout' term using case to distinguish, from Spanky's
rpatterson authored
70 While built on Python, some of the core services Buildout provides are
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
71 Python specific, and Buildout is most useful on Python projects, there
72 is nothing necessarily Python specific about using what a buildout
73 deploys and it can be used for non-Python projects.
74
75
008f160 @rpatterson Clarify the 'buildout' term using case to distinguish, from Spanky's
rpatterson authored
76 'Buildout' as a Technical Term
77 ==============================
78
79 A common source of confusion is the fact that 'buildout' as a term can
80 apply to two things that a user of Buildout will frequently have to
81 deal with. The first meaning of 'Buildout' is to refer to
82 `zc.buildout` the software. This includes the `bin/buildout` script
83 or when referring to running 'Buildout'. The other meaning is to
84 refer to the individual deployment directory as 'a buildout'. IOW,
85 when you checkout from version control into a working copy directory
86 that uses 'Buildout' the software, that directory is 'a buildout'. To
87 limit confusion about this, this tutorial uses 'Buildout' with a
88 capital 'B' to refer to the software and 'a buildout' with a lower
89 case 'b' to refer to a specific copy of a buildout in a specific
90 directory.
91
92
93 A Simple buildout
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
94 =================
95
ae4d2c9 @rpatterson Clarify using feeback from Spanky
rpatterson authored
96 A buildout configuration describes a deployment. This description is
97 written in configuration files, named with a `.cfg` extension by
98 convention. These files use an extended `ConfigParser
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
99 <http://docs.python.org/library/configparser.html>`_ format from the
ae4d2c9 @rpatterson Clarify using feeback from Spanky
rpatterson authored
100 Python standard library. You'll likely recognize this format as a
101 very common configuration file format consisting of named 'sections',
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
102 defined by a line with the section name in brackets. These sections
103 then contain named variables with values.
104
105 The core configuration of any buildout deployment is described in the
106 `[buildout]` section. A buildout deployment consists of 'parts',
4eb253a @rpatterson Simplify from Spanky
rpatterson authored
107 which are special configuration sections. The default configuration
108 file is `buildout.cfg`. As such, to create the simplest possible
109 buildout, which is an empty deployment, create an empty directory and
110 put the following into a `buildout.cfg` file in that directory::
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
111
112 [buildout]
113 parts =
114
115 Now that the deployment is described in the configuration file we can
008f160 @rpatterson Clarify the 'buildout' term using case to distinguish, from Spanky's
rpatterson authored
116 use Buildout to deploy the empty environment described. Deploying a
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
117 buildout has two steps. The first step defines which Python
7207e68 @rpatterson Clarify isolation aspects of the bootstrap step, from Spanky
rpatterson authored
118 installation to use for that buildout and establishes the isolated
119 Python environment described in `Why Buildout`_. This step also gets
008f160 @rpatterson Clarify the 'buildout' term using case to distinguish, from Spanky's
rpatterson authored
120 the minimum requirements necessary to use the Buildout software
7207e68 @rpatterson Clarify isolation aspects of the bootstrap step, from Spanky
rpatterson authored
121 itself. This step is called bootstrapping and is only necessary the
80036f0 @rpatterson Clarify that bootstrapping also needs to happen again if using a
rpatterson authored
122 first time a given copy of a given buildout is set up, if an
123 existing buildout is moved, or if a different Python installation is
124 used.
7207e68 @rpatterson Clarify isolation aspects of the bootstrap step, from Spanky
rpatterson authored
125
126 To bootstrap, copy into the buildout directory the `bootstrap.py`
127 script from::
128
129 http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
130
131 To establish the isolated Python environment, use the Python
132 installation that should be used for the buildout to run the
b20d00d @rpatterson Clarify why we use the `bootstrap.py -d` option
rpatterson authored
133 `bootstrap.py` script. In my opinion, it is best to use the `-d`
134 option with the `bootstrap.py` script so that Buildout will use the
135 more actively maintained `distribute` project. From that point on,
136 Buildout will use that Python installation for all subsequent Buildout
137 operations::
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
138
139 $ /path/to/python bootstrap.py -d
140 Creating directory '/opt/src/buildout-tutorial/bin'.
141 Creating directory '/opt/src/buildout-tutorial/parts'.
142 Creating directory '/opt/src/buildout-tutorial/develop-eggs'.
143 Generated script '/opt/src/buildout-tutorial/bin/Buildout'.
144
3eb2b50 @rpatterson Clarify some of what the bootstrap step does and set up the next chap…
rpatterson authored
145 Now the directory has three new directories, whose purposes will be
146 come clear in the next chapter, and a `bin/buildout` script (or
147 `bin\buildout.exe` on Windows) which is used to apply the deployment
148 configuration. Since this buildout configuration describes an empty
008f160 @rpatterson Clarify the 'buildout' term using case to distinguish, from Spanky's
rpatterson authored
149 deployment, running Buildout does nothing::
872bf53 @rpatterson Begin drafting a new, more generalized buildout tutorial
rpatterson authored
150
151 $ bin/buildout
3eb2b50 @rpatterson Clarify some of what the bootstrap step does and set up the next chap…
rpatterson authored
152
153 Next, we'll move beyond an empty buildout to an example of the
154 simplest possible buildout that actually deploys something.
b00062e @rpatterson Begin drafting the simplest buildout with a single Python distribution
rpatterson authored
155
156
157 The Simplest buildout
158 =====================
159
160 The simplest buildout will deploy a Python distribution in an isolated
86182d8 @rpatterson Switch to sphinx instead of distutils since the latter doesn't use
rpatterson authored
161 environment. In this case we'll add the `sphinx` distribution to
b00062e @rpatterson Begin drafting the simplest buildout with a single Python distribution
rpatterson authored
162 our empty buildout and Buildout will retrieve that distribution, build
163 it, install it isolated in the buildout, and add any console scripts
164 to the `bin` directory.
165
166 We tell Buildout what the pieces of a deployment are by adding special
167 sections to the configuration called 'parts'. Since deployments often
168 need to do many different kinds of things in the same deployment,
169 different parts need to be able to use different variables as options
170 and perform different logic and actions. As such, Buildout uses
171 different Python code for different kinds of parts to provide specific
172 deployment behavior. The Python code that handles a given buildout
173 part is called a `recipe`.
174
175 In the configuration file, a 'part' is just a named section that
176 provides a `recipe` variable, and whose section name is listed in the
177 `[buildout]` section's `parts` variable::
178
179 [buildout]
86182d8 @rpatterson Switch to sphinx instead of distutils since the latter doesn't use
rpatterson authored
180 parts = sphinx
b00062e @rpatterson Begin drafting the simplest buildout with a single Python distribution
rpatterson authored
181
86182d8 @rpatterson Switch to sphinx instead of distutils since the latter doesn't use
rpatterson authored
182 [sphinx]
0b755b7 @rpatterson Can use the default recipe now that we're using a dist that has conso…
rpatterson authored
183 recipe = zc.recipe.egg
b00062e @rpatterson Begin drafting the simplest buildout with a single Python distribution
rpatterson authored
184
185 In this case, we use the `zc.recipe.egg` recipe which is a part of the
186 Buildout project itself. This recipe retrieves Python
187 distributions, installs them isolated to the buildout, and also
188 handles installing console scripts. Later, we'll use part variables
189 as options to control the behavior of the recipe, but for now we'll
190 make use of the default behavior of `zc.recipe.egg` which is to get
191 the name of a single distribution to install from the part name.
192
193 Since we have already bootstrapped the buildout, haven't moved the
194 buildout directory, and we're using the same python, we do not need to
195 run the `bootstrap.py` script again. We can just update our buildout
196 by re-running `bin/buildout`::
197
198 $ bin/buildout
199 Getting distribution for 'zc.recipe.egg'.
200 Got zc.recipe.egg 1.3.2.
86182d8 @rpatterson Switch to sphinx instead of distutils since the latter doesn't use
rpatterson authored
201 Installing sphinx.
202 Getting distribution for 'sphinx'.
203 Got Sphinx 1.1.3.
204 Getting distribution for 'docutils>=0.7'.
9063a7f @rpatterson Omit setuptools warnings and apologize
rpatterson authored
205 warning: ...
b00062e @rpatterson Begin drafting the simplest buildout with a single Python distribution
rpatterson authored
206 zip_safe flag not set; analyzing archive contents...
9063a7f @rpatterson Omit setuptools warnings and apologize
rpatterson authored
207 docutils.parsers.rst.directives.misc: module references __file__...
b00062e @rpatterson Begin drafting the simplest buildout with a single Python distribution
rpatterson authored
208 Got docutils 0.8.1.
86182d8 @rpatterson Switch to sphinx instead of distutils since the latter doesn't use
rpatterson authored
209 Getting distribution for 'Jinja2>=2.3'.
9063a7f @rpatterson Omit setuptools warnings and apologize
rpatterson authored
210 warning: ...
86182d8 @rpatterson Switch to sphinx instead of distutils since the latter doesn't use
rpatterson authored
211 Got Jinja2 2.6.
212 Getting distribution for 'Pygments>=1.2'.
213 Got Pygments 1.5.
214 Generated script '/opt/src/buildout-tutorial/bin/sphinx-apidoc'.
215 Generated script '/opt/src/buildout-tutorial/bin/sphinx-build'.
216 Generated script '/opt/src/buildout-tutorial/bin/sphinx-quickstart'.
217 Generated script '/opt/src/buildout-tutorial/bin/sphinx-autogen'.
9063a7f @rpatterson Omit setuptools warnings and apologize
rpatterson authored
218
8020528 @rpatterson Finish describing what the simplest buildout does
rpatterson authored
219 Buildout tells us a bit about what it did while updating the
220 deployment to add the new part. It retrieved, built, and installed
221 the Python distributions for the recipe, the distribution required by
222 the part, and all of their dependencies. Note that it also reports
223 the versions it chose for the distributions it retrieved. We'll
224 discuss how to specify and control those versions later. Finally, it
225 installs the `console_scripts` specified in the `setup.py` of the
226 distribution specified in the part.
227
9063a7f @rpatterson Omit setuptools warnings and apologize
rpatterson authored
228 We've omitted some of the output that comes from building the eggs.
229 For context, that output most often occurs when building as
230 `setuptools` eggs Python distributions which only use Python's
231 `distutils`. The warnings come from distribute and are often not
232 important but do on occasion indicate a genuine problem.
233 Unfortunately, there's no clear way for a user who isn't an expert in
234 Buildout and distribute to interpret whether or not there is a
8020528 @rpatterson Finish describing what the simplest buildout does
rpatterson authored
235 problem. With apologies, the best answer is to ignore such messages
236 until you have reason to think there is a problem.
237
238 At this point we can safely run the sphinx console scripts in an
239 isolated evnironment::
240
241 $ bin/sphinx-apidoc --help
242 Usage: sphinx-apidoc [options] -o <output_path> <module_path> [exclude_paths, ...]
243
244 Look recursively in <module_path> for Python modules and packages and create
245 one reST file with automodule directives per package in the <output_path>...
cdb1ad0 @rpatterson Begin describing part options, need find a use-case for establishing
rpatterson authored
246
247
248 Section Variables and Part Options
249 ==================================
250
251 So far, this buildout is just being used as an isolated distribution
252 installer, but a reproducible deployment is more often much more than
253 that. Buildout is most useful for capturing and documenting the
254 specific details required by the deployment. Recipes are responsible
255 for supporting those details and expect to be given options to that
256 end. In the configuration file, those options are just the section
257 variables in the part that are recognized by the recipe.
258
259 The `zc.recipe.egg` recipe, for example, has a `dependent-scripts
260 <http://pypi.python.org/pypi/zc.recipe.egg/1.3.2#script-generation>`_
261 option. If `true`, this option causes the recipe to install the
262 console scripts for any of the distribution's dependencies that define
263 console scripts::
264
265 [buildout]
266 parts = sphinx
267
268 [sphinx]
269 recipe = zc.recipe.egg
270 dependent-scripts = true
271
272 Now when `bin/buildout` is run, the `pygmentize` console script from
273 the `Pygments` dependency of `sphinx` is also installed::
274
275 $ bin/buildout
276 Uninstalling sphinx.
277 Installing sphinx.
278 Generated script '/opt/src/buildout-tutorial/bin/sphinx-apidoc'.
279 Generated script '/opt/src/buildout-tutorial/bin/sphinx-build'.
280 Generated script '/opt/src/buildout-tutorial/bin/sphinx-quickstart'.
281 Generated script '/opt/src/buildout-tutorial/bin/sphinx-autogen'.
282 Generated script '/opt/src/buildout-tutorial/bin/pygmentize'.
283 $ bin/pygmentize --help
284 Usage: bin/pygmentize [-l <lexer> | -g] [-F <filter>[:<options>]] [-f <formatter>]
285 [-O <options>] [-P <option=value>] [-o <outfile>] [<infile>]...
Something went wrong with that request. Please try again.