mystack.py

mystack.py

Script format

As the low-level stringing-together build mechanism, we have an API that allows writing Python scripts like this ("mystack.py"):

import hashdist

system = hashdist.HostSystemPackageProvider()
qsnake = hashdist.SpkgPackageProvider('/home/dagss/qsnake/spkgs')
pypi = hashdist.PyPIPackageProvider('http://pypi.python.org')

# TBD: An environment mechanism like waf and scons, where one can
# reuse options

blas = spkg.reference_blas()
# or: blas = spkg.ATLAS(version='3.8.4')
# or: blas = system.ATLAS(version='3.8.4', libpath='/sysadmins/stupid/path/for/ATLAS')

intel = system.intel_compilers()
gcc = system.gcc_compilers()

python = system.python()
petsc = qsnake.petsc(blas=blas, compiler=intel)
petsc4py = qsnake.petsc4py(petsc=petsc, compiler=gcc, python=python )
numpy = spkg.numpy(python=python, blas=blas, compiler=intel, CFLAGS='-O0')
jinja2 = pypi.jinja2(python=python)

profile = hashdist.profile([python, petsc, numpy, jinja2])
hashdist.command_line(profile, distro_dir='/home/dagss/qsnake')

Note that this scripts embed a specific, chosen configuration. For a final distribution system, this file would either be generated, or just the same actions would be specified from higher-level configuration files. However, this format allows for rapid development and testing.

For Python-HPCMP, for instance, one could have one such script per cluster configuration to replace the current setup.

Script commands

The script can be invoked like this:

python mystack.py shell

to get a shell with the profile prefixed to PATH.

Command list:

• shell - Described above
• bacspec numpy - Get the spec.json to build numpy with BAC using this configuration
• debug numpy - Get a shell that drops into the build directory after BAC has done its unpacking/copying/environment setup; then echo the commands that would have been run to the console; and ask the user to investigate