mystack.py

mystack.py

We want to allow writing Python scripts ("mystack.py") which explicitly describes a full software stack.

The idea is to provide a Python package (hashdist) which has an API for building software from various sources. The API is used declarative (lazily) to describe the software dependency tree and build options, and then the BAC is invoked with the right options.

This mechanism alone might be sufficient when all the distribution users are power users, but in many cases another level of automation will be needed on top. The API should be written so that additional layers on top are encouraged and can compete for mindshare.

For Python-HPCMP, for instance, one could have one such script per cluster configuration to replace the current setup.

Attempt 1

import hashdist

system = hashdist.HostSystemPackageProvider()
qsnake = hashdist.SpkgPackageProvider('/home/dagss/qsnake/spkgs')
pypi = hashdist.PyPIPackageProvider('http://pypi.python.org')

# TBD: An environment mechanism like waf and scons, where one can
# reuse options

blas = spkg.reference_blas()
# or: blas = spkg.ATLAS(version='3.8.4')
# or: blas = system.ATLAS(version='3.8.4', libpath='/sysadmins/stupid/path/for/ATLAS')

intel = system.intel_compilers()
gcc = system.gcc_compilers()

python = system.python()
petsc = qsnake.petsc(blas=blas, compiler=intel)
petsc4py = qsnake.petsc4py(petsc=petsc, compiler=gcc, python=python)
numpy = qsnake.numpy(python=python, blas=blas, compiler=intel, CFLAGS='-O0')
jinja2 = pypi.jinja2(python=python)

profile = hashdist.profile([python, petsc, numpy, jinja2])
hashdist.command_line(profile, distro_dir='/home/dagss/qsnake')

Attempt 2

import hashdist

# the package environment has a list of sources to consider for packages;
# which will be searched in the order provided
env = hashdist.PackageEnvironment(sources=[
    hashdist.SystemPackageProvider('system'),
    hashdist.SpkgPackageProvider('qsnake', '/home/dagss/qsnake/spkgs'),
    hashdist.PyPIPackageProvider('pypi', 'http://pypi.python.org')
])

# The environment also stores default arguments. env is immutable, so we
# modify by making a copy
env = env.copy_with(CFLAGS=['-O0', '-g'])

# Set up some compilers; insist that they are found on the 'system' source
# (do not build them)
intel = env.pkgs.intel_compilers(from='system')
gcc = env.pkgs.gnu_compilers(from='system')

# env.pkgs.__getattr__ "instantiates" software. The result is simply a symbolic
# node in a build dependency graph; nothing is resolved until an actual build
# is invoked
blas = env.pkgs.reference_blas(compiler=intel)
# or: blas = env.pkgs.ATLAS(version='3.8.4', compiler=intel)
# or: blas = env.pkgs.ATLAS(version='3.8.4', from='system', libpath='/sysadmins/stupid/path/for/ATLAS')

python = env.pkgs.python()
petsc = env.pkgs.petsc(from='qsnake', blas=blas, compiler=intel)
petsc4py = env.pkgs.petsc4py(from='qsnake', petsc=petsc, compiler=gcc, python=python)
numpy = env.pkgs.numpy(python=python, blas=blas, compiler=intel, CFLAGS='-O2')
jinja2 = env.pkgs.jinja2(python=python)

profile = hashdist.profile([python, petsc, numpy, jinja2])
hashdist.command_line(profile, distro_dir='/home/dagss/qsnake')

Script commands

The script can be invoked like this:

python mystack.py shell

to get a shell with the profile prefixed to PATH.

Command list:

• shell - Described above
• bacspec numpy - Get the spec.json to build numpy with BAC using this configuration
• debug numpy - Get a shell that drops into the build directory after BAC has done its unpacking/copying/environment setup; then echo the commands that would have been run to the console; and ask the user to investigate