-
Notifications
You must be signed in to change notification settings - Fork 3
Buildit build system (deprecated)
License
mcdonc/buildit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Buildit Buildit makes it easier to create a repeatable deployment of software in a particular configuration. With it, you can perform conditional complilation of source code, install software, run scripts, or perform any repeatable sequence of tasks that ends up creating a known set files on your filesystem. On subsequent runs of the same set of tasks, Buildit performs the least amount of work possible to create the same set of files, only performing the work that it detects has not already been performed by earlier runs. Change History 0.1 -- Initial release 1.0 -- 4/15/2007, second release, see CHANGES.txt for info 1.1 -- (unreleased) Platforms Buildit runs under any platform on which Python supports the os.system command. This includes UNIX/Linux and Windows. It should be run via Python 2.4+. Installation Install Buildit by running the accompanying "setup.py" file, ala "python setup.py install". License The license under which Buildit is released is a BSD-like license. It can be found accompanying the distribution in the LICENSE.txt file. Rationale Buildit was created to allow me to write "buildout" profiles which need to perform arbitrary tasks in the service of setting up a client operating system environment with software and data specific to running applications I helped create. For instance, in one case, it is currently used to build multiple "instances" of Zope, ZEO, Apache/PHP, Squid, Python, and MySQL on a single machine, forming a software "profile" particular to that machine. The same buildout files are rearranged to create different instances of software on different machines at the same site. The same software is used to perform incremental upgrades to existing buildouts. Why Not Make? We had previously been using GNU Make for the same task, but my clients couldn't maintain the makefiles easily after the engagement ended because they were not willing to learn GNU Make's automatic variables and pattern rules which I used somewhat gratuitously to make my life easier. I also realized that even I could barely read the makefiles after I had been away from them for some time. Although make's fundamental behavior is very simple, it has a few problems. Because its basic feature set is so simple, and because it has been proven to need to do some reasonably complex things, many make versions have also accreted features over the years to allow for this complexity. Unfortunately, it was never meant to be a full-blown programming language, and the additions made to make syntax to support complex tasks are fairly mystifying. Additionally, if advanced make features are used within a Makefile, it is difficult to create, debug and maintain makefiles for those with little "make-fu". Even simple failing makefiles can be difficult to debug. Why Not Ant? I a big fan of neither Java nor hand-writing XML. Ant requires I use one and do the other. Why Not SCons? SCons is all about solving problems specific to the domain of source code recompilation. Buildit is much smaller, and more general. Why Not A-A-P? A-A-P was designed from the perspective of someone wanting to develop and install a software package on many different platforms. It does this very well, but loses some generality in the process. A-A-P also uses a make-like syntax for describing tasks, whereas Buildit uses Python. Why not zc.buildout? zc.buildout was released after Buildit was already mature. Additionally, zc.buildout appears to have a focus on Python eggs which Buildit does not. General Comparisons to Other Dependency Systems Buildit is, for better or worse, completely general and very simple. It performs OS shell tasks and calls in to arbitrary Python as necessary only as specified by the recipe-writer, rather than relying on any domain-specific implicit rules. Buildit includes no built-in provisions for building C/Java/C++/etc source to object code via implicit or user-defined pattern rules. In fact, it knows nothing whatsoever about creating software from source files into binaries. Unlike makefiles, Buildit "recipe" files have no intrinsic syntax. There are no tabs-vs-spaces issues, default rules, automatic variables, or any special kind of syntax at all. In Buildit, recipe files are defined within Python. If conditionals, looping, environment integration, or other advanced features become necessary within one of your recipes, rather than needing to spell these things within a special syntax, you just use Python instead. Unlike make, Buildit does not have the capability to perform parallel execution of tasks (although it will not prevent it from happening when it calls into make itself). Tasks A Buildit "task" is equivalent to a "rule" in GNUmake. It is the fundamental "unit of work" within buildit. A task has a name, a set of targets, a working directory, a set of commands, and a set of dependent tasks. These are described here. name -- A task's name is a freeform bit of text describing the purpose of the task. E.g. "configure python". Only one name may be provided for a task. A name is required. namespaces -- A task's namespaces name is a list or a string representing the namespace(s) to which this task belongs. "Local" references interpreted when executing the task will use replacement values from each namespace. targets -- The files that are created as a result of a successful run of this task. A task's targets are strings specifying the file that will be created as a result of this task. It may include buildit interpolation syntax (e.g. '${pkgdir}'), which will be resolved against the namespace set just while the task is performed. Relative target paths are considered relative to the workdir. workdir -- A task's workdir specifies the directory to which the OS will chdir before performing the commands implied by the task. Only one workdir may be specified. A workdir is optional, it needn't be specified in the task argument list. commands -- A task's command set is a list or tuple specifying the commands that do the work implied by the task, which, as a general rule, should involve creating the target file. The command set is typically a sequence of strings, although in addition to strings, special Python callable objects may be specified as a command. The strings that make up commands are resolved against the replacement dictionary for string interpolation. If only one string command is specified, it may be specified without embedding it in a list or a tuple (the same does not hold true for a single callable Python object used as a command, it must be embedded in a list or tuple). dependencies -- A task's dependency set is a sequence of other Task instances upon which this task depends. This is the way a dependency graph of tasks is formed. If only one dependency is specified, it may be specified without embedding it in a sequence. Task Example Here is an example task, which implies the work required to run 'configure' within a Python source tree:: configure = Task( 'configure python', namespaces = 'python', targets = '${sharedir}/build/${pkgdn}/Makefile', workdir = '${sharedir}', # we build Python using --enable-shared in order to allow plpython # to build against us on non-32-bit systems. It appears that this # isn't necessary on 32-bit systems (neither Linux nor Mac), but # required at least for x86_64 Linux. commands = [ "mkdir -p build/${pkgdn}", "cd build/${pkgdn} && ${sharedir}/src/${pkgdn}/configure \ --prefix=${sharedir}/opt/${pkgdn} --enable-shared", ], dependencies = (unpack, gcc4_patch_readline_c, py243_socket_patch) ) The Description The description of a task is just a string label. It is printed when Buildit is run to help you track down problems and give users a sense of what is happening when your recipes are run. It is required. In the above example, the description is 'configure python'. The Namespaces The namespaces of a task represent each namespace which it will attempt to use to resolve local names (e.g. ${./local} names). In the above example, the namespace is 'python'. If a task is provided a namespaces argument which is a single string with no spaces it it, it will be considered to have a single namespace. A task may have multiple namespaces. If a task has multiple namespaces, it will be executed once for each namespace in the list provided. For convenience, if a string with spaces in it is provided as the 'namespaces' attribute, it is parsed into a list of namespace names (this is mostly to work around the inability to define lists easily in ConfigParser format). The Targets The targets of a task are the files that are meant to be created by the commands specified within the task. Although the commands of a task may create many files and perform otherwise arbitrary actions, the target files are the files that must be created for Buildit itself to consider the task "complete". It may (and almost certainly will) require replacement interpolation. If only one target file is required, it can be specified as a string. If more than one target file is necessary, they must be supplied as strings within a Python sequence. We only have one target above. The target of our example above is '${sharedir}/build/${pkgdn}/Makefile'. A target file is not considered to be specified relative to the working directory: it must be an absolute path or must be specified relative to the current working directory from which the Buildit driver is invoked. However, it can contain interpolation syntax that will be resolved against the replacement object. A target is optional. If a task has no targets, it will be run unconditionally by Buildit on each invocation of the recipe in which it is contained. If all of a task's commands are run and the target files are not subsequently available on the filesystem, Buildit will throw an error. Buildit automatically "touches" target files after they've been created on the filesystem, so the date of all target files after a Buildit run will be close to "now", so there's no need to "touch" the target files manually. The Working Directory In the example task above, we specify a working directory ('${sharedir}'). The working directory indicates the directory into which we will tell the OS to chdir to before performing the commands indicated by the task. This is useful because it allows us to specify relative paths in commands which follow. When the task is finished, the working directory is unconditionally reset to the working directory that was effective before the task started. Task working directories take effect for only the duration of the task. Using a workdir is optional. If a workdir is not specified, the commands of the task will execute in the context of the working directory of the shell used to invoke the recipe file. The Commands In the example task shown above, we've specified two commands. The first one is:: "mkdir -p build/${pkgdn}" The second is:: "cd build/${pkgdn} && ${sharedir}/src/${pkgdn}/configure \ --prefix=${sharedir}/opt/${pkgdn} --enable-shared" Each command is a shell command. In this case, the shell commands are UNIX shell commands. The first command creates a build directory (in this case, relative to the workdir directory '${sharedir}'). The second changes the working directory to the newly-created build directory and runs the 'configure' script in the Python source tree with the "prefix" and "enable-shared" options. Note that each commands is interpolated against the namespace provided to the task. Thus if 'pkgdn' was 'Python-2.4.3' and 'sharedir' was '/tmp', the command would be expanded during execution like so:: mkdir -p build/Python-2.4.3 cd build/Python-2.4.3 && /tmp/src/Python-2.4.3/configure \ --prefix=/tmp/opt/Python-2.4.3 --enable-shared Note that the commands are executed serially in the order specified within the command set. Each command specified as a string is executed by Python's 'os.system'. If any command fails, (where "failure" is interpreted as a shell command exiting with a nonzero exit code), an error will be raised. Commands that aren't strings are assumed to be Python callables. This is not evident in the above example, but you may provide as a command a Python callable with a particular interface (see the commandlib module for examples). These kinds of commands are not executed by Python's 'os.system'; instead the callable is expected to do the work itself instead of delegating to the OS shell, although it is free to do whatever it needs to do (eg. the callable may do its own delegation to the OS shell if necessary). The Dependencies The dependency set of a task (specified by 'dependencies' in a Task constructor) identifies other Task instances upon which this task is dependent. "Is dependent" in the previous sentence means that the dependent task(s) must be completed before the task which declares it as a dependency may be run. Buildit has a simple algorithm for determining task and dependency "completeness" specified within "Task Recompletion Algorithm" later in this document. The dependency set of the example above is '(unpack, gcc4_patch_readline_c, py243_socket_patch)', which implies that the tasks named 'unpack', 'gcc4_patch_readline_c', and 'py243_socket_patch' must be completed before we can run the 'configure' task. The dependent tasks are not shown in the example, but for example, the 'unpack' task is presumably the task that places the Python source files into '${sharedir}/src/${pkgdn}'. A task needn't specify any dependencies. A task may specify a single dependency as a reference to a single Task instance, or it may specify a sequence of references to Task instances by embedding them in a list. Task Hints Tasks should be written with the expectation that they will be run more than once. For instance, if you create a symlink a directory within a command, the command should first check if a symlink already exists at that location, or you'll quite possibly end up symlinking the directory inside the existing symlinked directory on subsequent runs. Namespaces A Buildit namespace is a mapping of names to values. These mappings are used within tasks to perform textual variable replacement (which is also known as "interpolation"). Namespaces are user-defined. Multiple user-defined namespaces will typically exist during a given Buildit execution. All values in a given namespace are typically related to each other. For example, a 'squidinstance' namespace might represent all of the names and replacement values required to create an instance of the Squid proxy server. A 'pound' namespace might represent all of the names and replacement values required to create an installation of the Pound load balancer. A namespace is typically declared within one "section" of a Windows-style ".INI" file. Names within a namespace must consist solely of alphanumeric characters, the underscore, and the minus sign. The value for a name can be any set of characters and may also contain zero or more placeholders which mention other names that should be interpolated. These interpolation targets are known as "references", and they consist of a set of characters surrounded by squiggly brackets prefixed with a dollar sign (e.g. '${setofcharacters}'). Names and values are separated by any number of whitespace characters on either side of an equal sign. Here's an example of contents that might go into a namespace .INI file:: [anamespace] name1 = this is value one name2 = ${./name1} is relative to this namespace name3 = ${globalname} name4 = ${external/name} name5 = ${./name1} hello ${globalname} If you are examining the above example, you might note that there are four main types of strings which are allowed to compose a value: - String literals. In the above example, the string literal "this is value one" is assigned to the 'name1' name. - Rererences to "global" names, which are names which must be found in the "default" namespace. In the example above, the value of name 'name3' has a reference to the global name 'globalname'. Global names never have a slash character in them; they are always simple names without any prefixes. - References to "external" names, which are names that are found in other namespaces. In the example above, 'name4' refers to one external name, "${external/name}", which refers to the name 'name' in the external namespace named 'external'. External names always contain one slash, which separates the namespace name from the name that is to be looked up. If the 'external' namespace contained a name called 'name' with a value of 'foo', the external reference in the example would resolve to "foo". - References to "local" names, which are pointers to the values of names which are found in the same namespace as the name being defined. In the above example, the expansion of the value "${./name1} is relative to this namespace" in the local name 'name2' would become "this is value one is relative to this namespace". A "local" name always starts with the prefix "./". Essentially, local names are external names where the namespace name is ".". The Root .INI File In order to begin a Buildit project, first create a file named "root.ini" with the following content in Windows-style .INI format):: [globals] tgtdir = ${cwd}/sandbox [namespaces] foo = ${buildoutdir}/foo.ini [1.0] bar = ${buildoutdir}/bar/ini [1.0] baz = ${buildoutdir}/baz.ini It really doesn't matter what you name this file but for sake of reference let's say it's named "root.ini". Note that the file consists of two sections: a 'globals' section, and a 'namespaces' section. The 'globals' section allows you to define names and values which end up in the "global" namespace (see the 'Namespaces' section for a definition). The 'namespaces' section allows you to declare namespaces that will be used during the execution of Buildit. One or more lines may be defined within the namespaces section. Each line defines a namespace, and is composed of the following: - a name. In the above example, the namespaces 'foo' and 'bar' are declared. - a filename and an optional section name. In the above example, the section named "1.0" in the file named "${buildoutdir}/foo.ini" is used for the foo namespace. A space must separate the filename and the section name, and the section name must be surrounded by brackets. If no explicit section name is provided the file must contain a section named [default_namespace], which will be used. In all values within the root .INI file (the default values or the namespace values), you can use the following "built-in" global replacement values: ${cwd} -- the fullly-qualified path to the initial working directory of the process which invoked buildit. ${buildoutdir} -- the fullly-qualified path to the directory which contains the Python file that first invokes Buildit. ${username} -- the user name of the user who invoked the Python file that first invokes Buildit. ${platform} -- the value returned by distutils util.get_platform() function These names are also available in the default namespace when declaring values for other namespaces and running buildit tasks. The Namespace .INI Files Each name in the 'namespaces' section referred to within the "root' .INI file must exist on disk. Additionally, the section within the file that is mentioned on the value line must be contained within the named file. If your root .INI file declares the following namespace section:: [namespaces] breakfast = ${buildoutdir}/breakfast.ini [1.0] lunch = ${buildoutdir}/lunch.ini dinner = ${buildoutdir}/dinner.ini [1.0] .. then three additional .ini files need to exist on your filesystem, 'breakfast.ini', 'lunch.ini' and 'dinner.ini'. In general, these should live relative to the directory in which the python file which will initially invoke buildit (the "buildoutdir") lives. And in this case, 'breakfast.ini' and 'dinner.ini' both need to have a section named '1.0' which contains one or more key/value pairs that make up the namespace content. 'lunch.ini' must define a 'default_namespace' section since a section of that name is selected when no explicit section information is given. Without explaining much about what it means, here's an example of what might go in the "breakfast.ini" we've threatened to define above within our root .ini file:: [1.0] orderer = ${username} coffeesize = large coffeetype = espresso coffeeorder = ${./coffeesize} ${./coffeetype} bageltype = plain The "lunch.ini" file may look like this:: [default_namespace] orderer = ${username} coffeesize = small coffeetype = espresso coffeeorder = ${./coffeesize} ${./coffeetype} breadtype = rye bread Here's an additional example of what might go into the "dinner.ini" we've additionally threatened to define:: [1.0] orderer = ${username} coffeesize = small coffeetype = ${breakfast/coffeetype} coffeeorder = ${./coffeesize} ${./coffeetype} breadtype = dinner roll What's most interesting about what will "fall out" of these definitions is the result of the variable expansion. Again, without explaining why, assuming the current user name of the account running the Buildit process is "chrism", here's what the replacements would expand to in the 1.0 section of "breakfast.ini":: orderer = chrism coffeesize = large coffeetype = espresso coffeeorder = large espresso bageltype = plain The "lunch.ini" default_namespace section would expand to these values:: orderer = chrism coffeesize = small coffeetype = espresso coffeeorder = small espresso breadtype = rye bread And here's what the replacements would expand to in the 1.0 section of "dinner.ini":: orderer = chrism coffeesize = small coffeetype = espresso coffeeorder = small espresso breadtype = dinner roll During namespace processing, note that replacement targets can be replaced with global, local, or external values. It is an error to create two files which contain namespaces which depend on each other's names circularly, and it's an error to refer to a local, global, or external name that cannot be resolved because it does not exist. When Buildit is run, these types of errors are detected and presented to the person executing the Buildit script before any work is actually performed. Driving Buildit An example of kicking off a Buildit process:: # framework hair from buildit.context import Context from buildit.context import Software # your defined "tasks" from mytasks import mkbreakfast from mytasks import mkdinner # read default root .ini from file named "/etc/root.ini" and contextualize context = Context('/etc/root.ini') # use section 1.1 for dinner namespace instead of default named in root.ini context.set_section('dinner', '1.1') # use a different file and section for breakfast namespace instead of default context.set_file('breakfast', '${buildoutdir}/breakfast2.ini', 'coolbreakfast') # create a Software instance for both breakfast and dinner breakfast = Software(mkbreakfast, context) dinner = Software(mkdinner, context) # override the section value used for coffeetype and install breakfast.set('coffeetype', 'americano') breakfast.install() # override the section value used for breadtype and install dinner.set('breadtype', 'wheat') dinner.install() Driving Buildit More Declaratively via Config File "Instance" Sections Optionally, instead of using Python to drive buildit completely procedurally, you may choose to define sections within your root initialization file which represent software "instances" in the form (e.g.):: [breakfast:instance] buildit_task = mytasks.mkbreakfast buildit_order = 10 coffeetype = americano [dinner:instance] buildit_task = mytasks.mydinner buildit_order = 20 breadtype = wheat Section headers for instance definitions must end in ':instance'. The text that comes before ':instance' is purely informational. They must include a "buildit_task" value, which should be a Python dotted name identifier which points to a task instance. It can optionally include a "buildit_order" integer value, which represents the instance's execution order relative to the other instances defined. Lower numbered instances are run first. If an instance does not have a buildit_order, it is the same as providing it with a zero as a value. The example above does the same thing the procedural example does above, except we cannot substitute a different ini file or a different section name for the breakfast task (there is no analogue to set_file or set_section). All key/value pairs within the section which do not begin with "buildit_" are used as override variables for the namespace of the task named by the buildit_task dotted name. You can choose to put a namespace name after the dotted name value of buildit_task (e.g. 'buildit_task = mytasks.mydinner [breakfast]') to change the namespace in which these overrides will be performed. Once you've added instance sections, you can drive your buildout by using the boilerplate script (assuming /etc/root.ini is your config file):: from buildit.context import Context def main(root_ini): context = Context(root_ini) context.install() if __name__ == '__main__': import sys main('/etc/root.ini') Task Recompletion Algorithm A task is considered to be complete if all of the following statements can be made about it: - it specifies one or more target files in the task definition - all of its target files exist If a task does not meet these completion requirements at any given time, on a subsequent run of the recipe file in which it defined (or from a recipe file in which it is imported and used), its commands, *and all the commands of the tasks which are dependent upon it* will be rerun in dependency order. Buildit (unlike make) does not take into account the timestamp of a task's dependent targets when assessing whether a task needs to be recompleted. Command Library The buildit command library includes a number of standard command types that can be used in place of shell commands in the 'commands=' argument to a task. Each argument to the command can be a string literal or it can be a string which include expansion markers. These commands are available via 'from buildit.commandlib import X" where X includes: Download -- Download(filename, url, remove_on_error=True). CVSCheckout -- Checkout(repo, dir, module, tag=''). repo is the :ext: or :pserver: name of the repository including its path on the remote server's disk, dir is the directory into which we wish to check the module out, module is the CVS module name of the module, tag is the tag name, including the '-r '. Symlink -- Symlink(frm, to) Patch -- Patch(file, patch, level). 'patch' is the patchfle to apply to 'file', 'level' is the argument to apply to InFileWriter -- InFileWriter(infile, outfile, mode=0755). Replaces all <<HUGGED>> text in infile with an expansion based on the current namespace and the global namespace. Changes mode of outfile to mode. Substitute -- Substitute(filename, search_re, replacement_string, backupext='.~subst'). Replaces all text in 'filename' matching the regular expression string 'search_re' with the replacement_string. Make a backup of the original with the original filename plus the backup extension. SkelCopier -- SkelCopier(skeldir, tgtdir, destructive=''). Makes an exact copy of one directory ('skeldir') to another ('tgtdir'). If a file in the source directory has a .in extension, replace its <<HUGGED>> values with task interpolated values, and write it to the target directory without the .in extension. By default, the SkelCopier will not overwrite target files that already exist underneath tgtdir. If you pass the "destructive" argument with a string like 'yes' or 'true' the behavior will change and all existing target files will be overwritten. Reporting Bugs Please use "the buildit issue tracker":http://agendaless.com/Members/chrism/software/buildit_issues to report issues. Maillist The "buildit maillist:"http://lists.palladion.com/mailman/listinfo/buildit is where to discuss buildit-related issues. Have fun! Chris McDonough (chrism@agendaless.com)
About
Buildit build system (deprecated)
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published