Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
tree: 00118a1e84
Fetching contributors…

Cannot retrieve contributors at this time

724 lines (561 sloc) 29.939 kb
Buildit
Buildit makes it easier to create a repeatable deployment of
software in a particular configuration. With it, you can perform
conditional complilation of source code, install software, run
scripts, or perform any repeatable sequence of tasks that ends up
creating a known set files on your filesystem. On subsequent runs
of the same set of tasks, Buildit performs the least amount of work
possible to create the same set of files, only performing the work
that it detects has not already been performed by earlier runs.
Change History
0.1 -- Initial release
1.0 -- 4/15/2007, second release, see CHANGES.txt for info
1.1 -- (unreleased)
Platforms
Buildit runs under any platform on which Python supports the
os.system command. This includes UNIX/Linux and Windows. It should
be run via Python 2.4+.
Installation
Install Buildit by running the accompanying "setup.py" file, ala
"python setup.py install".
License
The license under which Buildit is released is a BSD-like license.
It can be found accompanying the distribution in the LICENSE.txt
file.
Rationale
Buildit was created to allow me to write "buildout" profiles which
need to perform arbitrary tasks in the service of setting up a
client operating system environment with software and data specific
to running applications I helped create. For instance, in one case,
it is currently used to build multiple "instances" of Zope, ZEO,
Apache/PHP, Squid, Python, and MySQL on a single machine, forming a
software "profile" particular to that machine. The same buildout
files are rearranged to create different instances of software on
different machines at the same site. The same software is used to
perform incremental upgrades to existing buildouts.
Why Not Make?
We had previously been using GNU Make for the same task, but my
clients couldn't maintain the makefiles easily after the
engagement ended because they were not willing to learn GNU Make's
automatic variables and pattern rules which I used somewhat
gratuitously to make my life easier. I also realized that even I
could barely read the makefiles after I had been away from them
for some time.
Although make's fundamental behavior is very simple, it has a few
problems. Because its basic feature set is so simple, and because
it has been proven to need to do some reasonably complex things,
many make versions have also accreted features over the years to
allow for this complexity. Unfortunately, it was never meant to
be a full-blown programming language, and the additions made to
make syntax to support complex tasks are fairly mystifying.
Additionally, if advanced make features are used within a
Makefile, it is difficult to create, debug and maintain makefiles
for those with little "make-fu". Even simple failing makefiles
can be difficult to debug.
Why Not Ant?
I a big fan of neither Java nor hand-writing XML. Ant requires I
use one and do the other.
Why Not SCons?
SCons is all about solving problems specific to the domain of
source code recompilation. Buildit is much smaller, and more
general.
Why Not A-A-P?
A-A-P was designed from the perspective of someone wanting to
develop and install a software package on many different
platforms. It does this very well, but loses some generality in
the process. A-A-P also uses a make-like syntax for describing
tasks, whereas Buildit uses Python.
Why not zc.buildout?
zc.buildout was released after Buildit was already mature.
Additionally, zc.buildout appears to have a focus on Python eggs
which Buildit does not.
General Comparisons to Other Dependency Systems
Buildit is, for better or worse, completely general and very simple.
It performs OS shell tasks and calls in to arbitrary Python as
necessary only as specified by the recipe-writer, rather than
relying on any domain-specific implicit rules.
Buildit includes no built-in provisions for building C/Java/C++/etc
source to object code via implicit or user-defined pattern rules.
In fact, it knows nothing whatsoever about creating software from
source files into binaries.
Unlike makefiles, Buildit "recipe" files have no intrinsic syntax.
There are no tabs-vs-spaces issues, default rules, automatic
variables, or any special kind of syntax at all. In Buildit, recipe
files are defined within Python. If conditionals, looping,
environment integration, or other advanced features become necessary
within one of your recipes, rather than needing to spell these
things within a special syntax, you just use Python instead.
Unlike make, Buildit does not have the capability to perform
parallel execution of tasks (although it will not prevent it from
happening when it calls into make itself).
Tasks
A Buildit "task" is equivalent to a "rule" in GNUmake. It is the
fundamental "unit of work" within buildit. A task has a name, a set
of targets, a working directory, a set of commands, and a set of
dependent tasks. These are described here.
name -- A task's name is a freeform bit of text describing the
purpose of the task. E.g. "configure python". Only one name may be
provided for a task. A name is required.
namespaces -- A task's namespaces name is a list or a string
representing the namespace(s) to which this task belongs. "Local"
references interpreted when executing the task will use replacement
values from each namespace.
targets -- The files that are created as a result of a successful
run of this task. A task's targets are strings specifying the file
that will be created as a result of this task. It may include
buildit interpolation syntax (e.g. '${pkgdir}'), which will be
resolved against the namespace set just while the task is performed.
Relative target paths are considered relative to the workdir.
workdir -- A task's workdir specifies the directory to which the OS
will chdir before performing the commands implied by the task. Only
one workdir may be specified. A workdir is optional, it needn't be
specified in the task argument list.
commands -- A task's command set is a list or tuple specifying the
commands that do the work implied by the task, which, as a general
rule, should involve creating the target file. The command set is
typically a sequence of strings, although in addition to strings,
special Python callable objects may be specified as a command. The
strings that make up commands are resolved against the replacement
dictionary for string interpolation. If only one string command is
specified, it may be specified without embedding it in a list or a
tuple (the same does not hold true for a single callable Python
object used as a command, it must be embedded in a list or tuple).
dependencies -- A task's dependency set is a sequence of other Task
instances upon which this task depends. This is the way a
dependency graph of tasks is formed. If only one dependency is
specified, it may be specified without embedding it in a sequence.
Task Example
Here is an example task, which implies the work required to run
'configure' within a Python source tree::
configure = Task(
'configure python',
namespaces = 'python',
targets = '${sharedir}/build/${pkgdn}/Makefile',
workdir = '${sharedir}',
# we build Python using --enable-shared in order to allow plpython
# to build against us on non-32-bit systems. It appears that this
# isn't necessary on 32-bit systems (neither Linux nor Mac), but
# required at least for x86_64 Linux.
commands = [
"mkdir -p build/${pkgdn}",
"cd build/${pkgdn} && ${sharedir}/src/${pkgdn}/configure \
--prefix=${sharedir}/opt/${pkgdn} --enable-shared",
],
dependencies = (unpack, gcc4_patch_readline_c, py243_socket_patch)
)
The Description
The description of a task is just a string label. It is printed
when Buildit is run to help you track down problems and give users
a sense of what is happening when your recipes are run. It is
required. In the above example, the description is 'configure
python'.
The Namespaces
The namespaces of a task represent each namespace which it will
attempt to use to resolve local names (e.g. ${./local} names). In
the above example, the namespace is 'python'.
If a task is provided a namespaces argument which is a single
string with no spaces it it, it will be considered to have a
single namespace.
A task may have multiple namespaces. If a task has multiple
namespaces, it will be executed once for each namespace in the
list provided. For convenience, if a string with spaces in it is
provided as the 'namespaces' attribute, it is parsed into a list
of namespace names (this is mostly to work around the inability to
define lists easily in ConfigParser format).
The Targets
The targets of a task are the files that are meant to be created
by the commands specified within the task. Although the commands
of a task may create many files and perform otherwise arbitrary
actions, the target files are the files that must be created for
Buildit itself to consider the task "complete". It may (and almost
certainly will) require replacement interpolation. If only one
target file is required, it can be specified as a string. If more
than one target file is necessary, they must be supplied as
strings within a Python sequence. We only have one target above.
The target of our example above is
'${sharedir}/build/${pkgdn}/Makefile'.
A target file is not considered to be specified relative to the
working directory: it must be an absolute path or must be
specified relative to the current working directory from which the
Buildit driver is invoked. However, it can contain interpolation
syntax that will be resolved against the replacement object.
A target is optional. If a task has no targets, it will be run
unconditionally by Buildit on each invocation of the recipe in
which it is contained.
If all of a task's commands are run and the target files are not
subsequently available on the filesystem, Buildit will throw an
error.
Buildit automatically "touches" target files after they've been
created on the filesystem, so the date of all target files after a
Buildit run will be close to "now", so there's no need to "touch"
the target files manually.
The Working Directory
In the example task above, we specify a working directory
('${sharedir}'). The working directory indicates the directory
into which we will tell the OS to chdir to before performing the
commands indicated by the task. This is useful because it allows
us to specify relative paths in commands which follow. When the
task is finished, the working directory is unconditionally reset
to the working directory that was effective before the task
started. Task working directories take effect for only the
duration of the task. Using a workdir is optional. If a workdir
is not specified, the commands of the task will execute in the
context of the working directory of the shell used to invoke the
recipe file.
The Commands
In the example task shown above, we've specified two
commands. The first one is::
"mkdir -p build/${pkgdn}"
The second
is::
"cd build/${pkgdn} && ${sharedir}/src/${pkgdn}/configure \
--prefix=${sharedir}/opt/${pkgdn} --enable-shared"
Each command is a shell command. In this case, the shell commands
are UNIX shell commands.
The first command creates a build directory (in this case,
relative to the workdir directory '${sharedir}'). The second
changes the working directory to the newly-created build directory
and runs the 'configure' script in the Python source tree with the
"prefix" and "enable-shared" options. Note that each commands is
interpolated against the namespace provided to the task. Thus if
'pkgdn' was 'Python-2.4.3' and 'sharedir' was '/tmp', the command
would be expanded during execution like so::
mkdir -p build/Python-2.4.3
cd build/Python-2.4.3 && /tmp/src/Python-2.4.3/configure \
--prefix=/tmp/opt/Python-2.4.3 --enable-shared
Note that the commands are executed serially in the order
specified within the command set. Each command specified as a
string is executed by Python's 'os.system'. If any command fails,
(where "failure" is interpreted as a shell command exiting with a
nonzero exit code), an error will be raised.
Commands that aren't strings are assumed to be Python callables.
This is not evident in the above example, but you may provide as a
command a Python callable with a particular interface (see the
commandlib module for examples). These kinds of commands are not
executed by Python's 'os.system'; instead the callable is expected
to do the work itself instead of delegating to the OS shell,
although it is free to do whatever it needs to do (eg. the
callable may do its own delegation to the OS shell if necessary).
The Dependencies
The dependency set of a task (specified by 'dependencies' in a
Task constructor) identifies other Task instances upon which this
task is dependent. "Is dependent" in the previous sentence means
that the dependent task(s) must be completed before the task which
declares it as a dependency may be run. Buildit has a simple
algorithm for determining task and dependency "completeness"
specified within "Task Recompletion Algorithm" later in this
document.
The dependency set of the example above is '(unpack,
gcc4_patch_readline_c, py243_socket_patch)', which implies that
the tasks named 'unpack', 'gcc4_patch_readline_c', and
'py243_socket_patch' must be completed before we can run the
'configure' task. The dependent tasks are not shown in the
example, but for example, the 'unpack' task is presumably the task
that places the Python source files into
'${sharedir}/src/${pkgdn}'.
A task needn't specify any dependencies. A task may specify a
single dependency as a reference to a single Task instance, or it
may specify a sequence of references to Task instances by
embedding them in a list.
Task Hints
Tasks should be written with the expectation that they will be run
more than once. For instance, if you create a symlink a directory
within a command, the command should first check if a symlink
already exists at that location, or you'll quite possibly end up
symlinking the directory inside the existing symlinked directory on
subsequent runs.
Namespaces
A Buildit namespace is a mapping of names to values. These mappings
are used within tasks to perform textual variable replacement (which
is also known as "interpolation").
Namespaces are user-defined. Multiple user-defined namespaces will
typically exist during a given Buildit execution. All values in a
given namespace are typically related to each other. For example, a
'squidinstance' namespace might represent all of the names and
replacement values required to create an instance of the Squid proxy
server. A 'pound' namespace might represent all of the names and
replacement values required to create an installation of the Pound
load balancer.
A namespace is typically declared within one "section" of a
Windows-style ".INI" file. Names within a namespace must consist
solely of alphanumeric characters, the underscore, and the minus
sign. The value for a name can be any set of characters and may
also contain zero or more placeholders which mention other names
that should be interpolated. These interpolation targets are known
as "references", and they consist of a set of characters surrounded
by squiggly brackets prefixed with a dollar sign
(e.g. '${setofcharacters}'). Names and values are separated by any
number of whitespace characters on either side of an equal sign.
Here's an example of contents that might go into a namespace .INI
file::
[anamespace]
name1 = this is value one
name2 = ${./name1} is relative to this namespace
name3 = ${globalname}
name4 = ${external/name}
name5 = ${./name1} hello ${globalname}
If you are examining the above example, you might note that there
are four main types of strings which are allowed to compose a
value:
- String literals. In the above example, the string literal "this
is value one" is assigned to the 'name1' name.
- Rererences to "global" names, which are names which must be found
in the "default" namespace. In the example above, the value of
name 'name3' has a reference to the global name 'globalname'.
Global names never have a slash character in them; they are
always simple names without any prefixes.
- References to "external" names, which are names that are found in
other namespaces. In the example above, 'name4' refers to one
external name, "${external/name}", which refers to the name
'name' in the external namespace named 'external'. External
names always contain one slash, which separates the namespace
name from the name that is to be looked up. If the 'external'
namespace contained a name called 'name' with a value of 'foo',
the external reference in the example would resolve to "foo".
- References to "local" names, which are pointers to the values of
names which are found in the same namespace as the name being
defined. In the above example, the expansion of the value
"${./name1} is relative to this namespace" in the local name
'name2' would become "this is value one is relative to this
namespace". A "local" name always starts with the prefix "./".
Essentially, local names are external names where the namespace
name is ".".
The Root .INI File
In order to begin a Buildit project, first create a file named
"root.ini" with the following content in Windows-style .INI
format)::
[globals]
tgtdir = ${cwd}/sandbox
[namespaces]
foo = ${buildoutdir}/foo.ini [1.0]
bar = ${buildoutdir}/bar/ini [1.0]
baz = ${buildoutdir}/baz.ini
It really doesn't matter what you name this file but for sake of
reference let's say it's named "root.ini".
Note that the file consists of two sections: a 'globals' section,
and a 'namespaces' section.
The 'globals' section allows you to define names and values which
end up in the "global" namespace (see the 'Namespaces' section for a
definition).
The 'namespaces' section allows you to declare namespaces that will
be used during the execution of Buildit. One or more lines may be
defined within the namespaces section. Each line defines a
namespace, and is composed of the following:
- a name. In the above example, the namespaces 'foo' and 'bar' are
declared.
- a filename and an optional section name. In the above example,
the section named "1.0" in the file named "${buildoutdir}/foo.ini"
is used for the foo namespace. A space must separate the filename
and the section name, and the section name must be surrounded by
brackets. If no explicit section name is provided the file must
contain a section named [default_namespace], which will be used.
In all values within the root .INI file (the default values or the
namespace values), you can use the following "built-in" global
replacement values:
${cwd} -- the fullly-qualified path to the initial working
directory of the process which invoked buildit.
${buildoutdir} -- the fullly-qualified path to the directory which
contains the Python file that first invokes Buildit.
${username} -- the user name of the user who invoked the Python
file that first invokes Buildit.
${platform} -- the value returned by distutils util.get_platform()
function
These names are also available in the default namespace when
declaring values for other namespaces and running buildit tasks.
The Namespace .INI Files
Each name in the 'namespaces' section referred to within the "root'
.INI file must exist on disk. Additionally, the section within the
file that is mentioned on the value line must be contained within
the named file.
If your root .INI file declares the following namespace section::
[namespaces]
breakfast = ${buildoutdir}/breakfast.ini [1.0]
lunch = ${buildoutdir}/lunch.ini
dinner = ${buildoutdir}/dinner.ini [1.0]
.. then three additional .ini files need to exist on your filesystem,
'breakfast.ini', 'lunch.ini' and 'dinner.ini'. In general, these
should live relative to the directory in which the python file which
will initially invoke buildit (the "buildoutdir") lives. And in this
case, 'breakfast.ini' and 'dinner.ini' both need to have a section
named '1.0' which contains one or more key/value pairs that make up
the namespace content. 'lunch.ini' must define a 'default_namespace'
section since a section of that name is selected when no explicit
section information is given.
Without explaining much about what it means, here's an example of
what might go in the "breakfast.ini" we've threatened to define
above within our root .ini file::
[1.0]
orderer = ${username}
coffeesize = large
coffeetype = espresso
coffeeorder = ${./coffeesize} ${./coffeetype}
bageltype = plain
The "lunch.ini" file may look like this::
[default_namespace]
orderer = ${username}
coffeesize = small
coffeetype = espresso
coffeeorder = ${./coffeesize} ${./coffeetype}
breadtype = rye bread
Here's an additional example of what might go into the "dinner.ini"
we've additionally threatened to define::
[1.0]
orderer = ${username}
coffeesize = small
coffeetype = ${breakfast/coffeetype}
coffeeorder = ${./coffeesize} ${./coffeetype}
breadtype = dinner roll
What's most interesting about what will "fall out" of these
definitions is the result of the variable expansion. Again, without
explaining why, assuming the current user name of the account
running the Buildit process is "chrism", here's what the
replacements would expand to in the 1.0 section of "breakfast.ini"::
orderer = chrism
coffeesize = large
coffeetype = espresso
coffeeorder = large espresso
bageltype = plain
The "lunch.ini" default_namespace section would expand to these
values::
orderer = chrism
coffeesize = small
coffeetype = espresso
coffeeorder = small espresso
breadtype = rye bread
And here's what the replacements would expand to in the 1.0 section
of "dinner.ini"::
orderer = chrism
coffeesize = small
coffeetype = espresso
coffeeorder = small espresso
breadtype = dinner roll
During namespace processing, note that replacement targets can be
replaced with global, local, or external values.
It is an error to create two files which contain namespaces which
depend on each other's names circularly, and it's an error to refer
to a local, global, or external name that cannot be resolved because
it does not exist. When Buildit is run, these types of errors are
detected and presented to the person executing the Buildit script
before any work is actually performed.
Driving Buildit
An example of kicking off a Buildit process::
# framework hair
from buildit.context import Context
from buildit.context import Software
# your defined "tasks"
from mytasks import mkbreakfast
from mytasks import mkdinner
# read default root .ini from file named "/etc/root.ini" and contextualize
context = Context('/etc/root.ini')
# use section 1.1 for dinner namespace instead of default named in root.ini
context.set_section('dinner', '1.1')
# use a different file and section for breakfast namespace instead of default
context.set_file('breakfast', '${buildoutdir}/breakfast2.ini',
'coolbreakfast')
# create a Software instance for both breakfast and dinner
breakfast = Software(mkbreakfast, context)
dinner = Software(mkdinner, context)
# override the section value used for coffeetype and install
breakfast.set('coffeetype', 'americano')
breakfast.install()
# override the section value used for breadtype and install
dinner.set('breadtype', 'wheat')
dinner.install()
Driving Buildit More Declaratively via Config File "Instance" Sections
Optionally, instead of using Python to drive buildit completely
procedurally, you may choose to define sections within your root
initialization file which represent software "instances" in the form
(e.g.)::
[breakfast:instance]
buildit_task = mytasks.mkbreakfast
buildit_order = 10
coffeetype = americano
[dinner:instance]
buildit_task = mytasks.mydinner
buildit_order = 20
breadtype = wheat
Section headers for instance definitions must end in ':instance'.
The text that comes before ':instance' is purely informational.
They must include a "buildit_task" value, which should be a Python
dotted name identifier which points to a task instance. It can
optionally include a "buildit_order" integer value, which represents
the instance's execution order relative to the other instances
defined. Lower numbered instances are run first. If an instance
does not have a buildit_order, it is the same as providing it with a
zero as a value.
The example above does the same thing the procedural example does
above, except we cannot substitute a different ini file or a
different section name for the breakfast task (there is no analogue
to set_file or set_section). All key/value pairs within the section
which do not begin with "buildit_" are used as override variables
for the namespace of the task named by the buildit_task dotted name.
You can choose to put a namespace name after the dotted name value
of buildit_task (e.g. 'buildit_task = mytasks.mydinner [breakfast]')
to change the namespace in which these overrides will be performed.
Once you've added instance sections, you can drive your buildout by
using the boilerplate script (assuming /etc/root.ini is your config
file)::
from buildit.context import Context
def main(root_ini):
context = Context(root_ini)
context.install()
if __name__ == '__main__':
import sys
main('/etc/root.ini')
Task Recompletion Algorithm
A task is considered to be complete if all of the following
statements can be made about it:
- it specifies one or more target files in the task definition
- all of its target files exist
If a task does not meet these completion requirements at any given
time, on a subsequent run of the recipe file in which it defined (or
from a recipe file in which it is imported and used), its commands,
*and all the commands of the tasks which are dependent upon it* will
be rerun in dependency order.
Buildit (unlike make) does not take into account the timestamp of a
task's dependent targets when assessing whether a task needs to be
recompleted.
Command Library
The buildit command library includes a number of standard command
types that can be used in place of shell commands in the 'commands='
argument to a task. Each argument to the command can be a string
literal or it can be a string which include expansion markers.
These commands are available via 'from buildit.commandlib import X"
where X includes:
Download -- Download(filename, url, remove_on_error=True).
CVSCheckout -- Checkout(repo, dir, module, tag=''). repo is the
:ext: or :pserver: name of the repository including its path on the
remote server's disk, dir is the directory into which we wish to
check the module out, module is the CVS module name of the module,
tag is the tag name, including the '-r '.
Symlink -- Symlink(frm, to)
Patch -- Patch(file, patch, level). 'patch' is the patchfle to
apply to 'file', 'level' is the argument to apply to
InFileWriter -- InFileWriter(infile, outfile, mode=0755). Replaces
all <<HUGGED>> text in infile with an expansion based on the current
namespace and the global namespace. Changes mode of outfile to
mode.
Substitute -- Substitute(filename, search_re, replacement_string,
backupext='.~subst'). Replaces all text in 'filename' matching the
regular expression string 'search_re' with the replacement_string.
Make a backup of the original with the original filename plus the
backup extension.
SkelCopier -- SkelCopier(skeldir, tgtdir, destructive=''). Makes an
exact copy of one directory ('skeldir') to another ('tgtdir'). If a
file in the source directory has a .in extension, replace its
<<HUGGED>> values with task interpolated values, and write it to the
target directory without the .in extension. By default, the SkelCopier
will not overwrite target files that already exist underneath tgtdir.
If you pass the "destructive" argument with a string like 'yes' or
'true' the behavior will change and all existing target files will be
overwritten.
Reporting Bugs
Please use "the buildit issue
tracker":http://agendaless.com/Members/chrism/software/buildit_issues
to report issues.
Maillist
The "buildit
maillist:"http://lists.palladion.com/mailman/listinfo/buildit is
where to discuss buildit-related issues.
Have fun!
Chris McDonough (chrism@agendaless.com)
Jump to Line
Something went wrong with that request. Please try again.