BAC

Build Artifact Cache

bac is a command-line tool written in Python that is responsible for managing the build artifact cache directory. The most important aspect of BAC is what it does not do, but leave as the responsibility of the caller.

BAC is a tool which is only used by package management systems (or a package developer during debugging).

The BAC store

The BAC store is a directory structure on disk that looks roughly like this:

$bacstore/sources/python-2.7.1.tar.gz  # cached downloaded tarball
$bacstore/sources/numpy.git            # raw git repository
$bacstore/sourcedb/hKUWhBuneltGSN4s0N-LMOpG27Q # see bac fetch command

$bacstore/builds/python-hvfkN-qlp-zhXR3cuerq6jd2Z7g/bin/python # already produces builds
$bacstore/builds/...

$backstore/tmp/numpy-6dcfXufJLW3J6S-9rRe4vUlBj5g/.... # build in progress or failed

Source fetching

The first feature of BAC is managing downloading source code and labeling it, so that it can be referred to in build instructions:

bac fetch /path/to/bacstore http://python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2 [optional md5 hash]
bac fetch /path/to/bacstore https://github.com/numpy/numpy.git 72185d34170369ec07e8e84ed18d2f6a814e327a

Each of these commands will

• Download the given sources to $backstore/sources (this may be a wget or a git clone or a git remote add ...; git fetch ...)
• Make a label (such as rDR41po8gfpi5g9cNpYWWk5easQ)
• Create a database file in $bacstore/sourcedb/$label containing the location of the downloaded source (i.e. a tarfile; or a git repo + a git commit)

The sister command takes a label for a set of sources and unpacks them:

bac unpack rDR41po8gfpi5g9cNpYWWk5easQ [dest-dir]

(be aware of the git archive command when implementing this).

Building artifacts

The following command,:

bac build /path/to/bacstore mybuild.json

ensures that the build specified by the mybuild.json is available in the store (by building it if necesarry), and returns its location. The build specification file is described below.

On success, returns (on standard output) the resulting path to the built package. This could have been built on the fly, or been found already existing in the cache. (Some status information should be provided as one goes too so that one can tail logfiles etc. too; all easily parseable by a scripted called).

In the event of a failure, a directory will be left in $bacstore/tmp that can be inspected for post-mortem debugging. It is the responsibility of the caller to remove this.

bac check /path/to/bacstore mybuild.json

Checks if the build is already present

bac debug /path/to/bacstore mybuild.json

Like build, but only goes through setting up the environment for the build, then echoes the commands that it would have executed had a build been requested, then drops into a shell.

More commands that should be the responsibility of BAC:

• Garbage collection

Build specification

The build specification fully describes a build, so that it can be hashed. Example:

{ 'dependencies' :
    { 'numpy' : 'numpy-345wr23wrfw3r4w',
      'blas' : 'ATLAS-32rasdfasdfasdf',
      'gcc' : 'gcc-324qwed32e2q3d',
      'python' : 'python-q324rfaewfcwqrf',
      'bash' : 'bash-34raewfvq23rw'
    },
  'copy-files' : ['/home/dagss/qsnake/foo/spkgs/mypackage.install'],
  'files' : {
      'filename' : 'mypackage.ini',
      'contents' : [
          'somearg = foo'
      ]},
  'sources' : '6dcfXufJLW3J6S-9rRe4vUlBj5g',
  'build' :
      { 'interpreter' : '$bash/bin/bash --someflag $1',
        'script' : [
            'source $gcc/build_env',
            'export USE_FROBNICATOR=True',
            'export CFLAGS=-O3',
            'bash mypackage.install']
     },
}

(Note: lists of strings are multi-line strings encoded as json)

name

Will be prepended to the hash string in the cache (for human-readability).

dependencies

Specifies existing build artifacts that is depended upon by this build. If these are not present already, fail. The full expanded list of dependencies should be present (it is not known that numpy once upon a time depended on Python). Each dependency will be present in the environment variables when running the build script. These strings are included in the hash.

copy-files

Lists files reachable on the local filesystem. The contents of these files are included in the hash, and the file copied to the build directory.

files

Files that should be created verbatim in the build directory.

sources

The label for the tarball or git commit for the sources that should be built (as passed to bac unpack). This must have been previously downloaded using bac fetch.

build/interpreter

The interpreter command that is used (including arguments). Can be a command to be looked up in the environment PATH (bash), or a full path (starts with /), or it can start with $ and one of the keys of the dependencies dict, in which case one can select a binary from a previously built artifact.

build/script

Simply the script given to the interpreter. If $1 is found in the interpreter command line, this is saved to file and passed as that argument. Otherwise, feed it on standard input.