Use Spack as a build system

[vkarak]

It would be nice to be able to use Spack as a build system for building the source code of a test.

This relates to #300.

[sjpb]

And also as a module system - yes you can use spack's generated modules but in some ways using "spack load " directly would be more convenient and allow you to use spack specs in environment definitions.

[vkarak]

@victorusu Any ideas on this one?

[victorusu]

The way I see this task is by separated into two orthogonal issues:

1. Install software with Spack
2. use the software installed with Spack

I see them as orthogonal because the other build systems (supported by ReFrame) do not have mechanisms to "load" the installed software. In fact, they can only be used to build software. I think we should keep them separated. But I may be wrong because Spack does more than just building software. We may have an opportunity to design a holistic solution for Spack if we combine both issues.

All the mechanisms to install software using Spack can be called using the srun command as shown in the example below

srun --nodes=5 --ntasks-per-node=3 --cpus-per-task=4 /path/to/spack install -j4

This would effectively spawn 15 Spack instances across 3 nodes and each could use 4 CPUs to build each package. That could potentially be used to install 15 independent packages from a build tree.

Currently, ReFrame doesn't support to emit build commands using a job launcher, which again, I think that this is another orthogonal issue.

Install software with Spack

Spack, for now, has the possibility to have different ways to install a package. One can use

• spack install
• spack install # in a folder that has a spack.yaml or a spack.lock file defined

Using spack install

The first one is more "traditional" and the user can select different configurations (e.g. installation directories, modules creation) by defining configuration folders using custom scopes the flag -C.
It would look like this

spack -C <path-to-config-folder> install gromacs +mpi ^mpich@3.2.1 +slurm +argobots ^fftw +mpi +openmp precision=long_double,double,quad,float

The associated documentation from Spack for spack install and for the custom scopes.

Using Spack Environments

The second one allows one to define what it is called an environment. This is a similar concept to Python's virtual environments. And reflects more or less how Nix deploy their software, especially if one uses environment views. A description of Nix for those interested can be found here and a TL;DR from Compute Canada.

Spack Environments are very flexible because one defines what to be installed using a yaml file, which combines the configuration and the package specification. This environment is a very powerful concept with many options.

In the example below I showing the usage of Spack Environments to install the MPI version of GROMACS with MPICH and FFTW, despite the fact that my default configuration dictates that I should be using OpenMPI and intel-mkl by default.

spack:
    specs:
      - gromacs +mpi ^fftw
      - mpich@3.2.1 +slurm +argobots
      - fftw +mpi +openmp precision=long_double,double,quad,float 
    concretization: together

The option concretization: together combines the instructions to all specs together. This effectively makes GROMACS to be installed using the spec

gromacs +mpi ^mpich@3.2.1 +slurm +argobots ^fftw +mpi +openmp precision=long_double,double,quad,float 

One can write the yaml file as

spack:
    specs:
      - gromacs@2020.4 ~mpi %gcc@10.2 ^fftw ~mpi 
    concretization: together
    config:
        install_tree: <path-to-installation-dir>
        shared_linking: runpath # instead of the default rpath from spack
        install_missing_compilers: true

which would effectively install GROMACS version 2020.4 without MPI using FFTW compiled without MPI both using GCC version 10.2. The concretization: together has no effect, but the config section sets the installation dir to be <path-to-installation-dir>, tell Spack to install GCC 10.2 if it was previously detected, and links all the libraries using the RUNPATH mechanism.

Spack Environments allow the concept of Spec Matrixes, which basically allows the user to create (almost) complete software stacks using different compiler and library combinations. The almost in parentheses is just because software stacks can be very complex and sometimes they cannot be expressed in matrixes. Nevertheless, the matrix syntax is quite powerful and allows the user to set spec constraints.

Use software installed with Spack

AFAIK, depending on how the software was installed, one may or may not be able to use spack load.
If one has a software stack built using Spack one can replace the module load command with a spack load one.

IMHO, this is a completely different issue. We should support this but on a different issue - dedicated to extending the ModuleSystem to include spack load.

AFAIK, depending on the configuration of the Spack Environment used to build the software, one may need to activate the environment before being able to use the installed software. I can provide more details about this if needed, but one may be able to see the installed software of an environment.

[victorusu]

Having posted the previous comment, I would like to propose to start slowly, by introducing Sapck through the spack install <pkg-spec> command, which is option number one described above.
This would allow us to properly see how spack gets used and its limitations in the wild.

Unless, of course, someone already has a use case! 😄

My Spack test cases are a list of dependency tests, where the root test downloads Spack using

self.sourcesdir = 'https://github.com/spack/spack.git'

and configures Spack using run_after('setup') hooks

    @rfm.run_after('setup')
    def config_spack(self):
        self.spack_path = self.stagedir
        self.variables = {
            'SPACK_ROOT': f'{self.spack_path}',
            'PATH': '$SPACK_ROOT/bin:$PATH',
        }

        self.prebuild_cmds += [
            f'spack compiler find --scope site',
            # I wish I could be using "external find", but I cannot because it breaks my setup
            # since external find ALWAYS write to ~/.spack/packages.yaml! Why??
            # f'spack external find',
            # removing recursive link ($spack/lib/spack/docs/_spack_root) from spack folder
            f'rm -rf {self.spack_path}/lib/spack/docs'
        ]
        # omitted the part that creates the `modules.yaml` file using pure Python to have proper module names for the installed packages.
        ...

The next dependent ReFrame tests install each package using

    def __init__(self, pkg, spec):
        self.executable = 'spack'
        self.executable_opts = ['install', f'{pkg}', f'{spec}']
        ...

with a run_after('setup') hook the build to Spack

    @rfm.run_after('setup')
    def config_spack(self):
        target = self.getdep(self.dep_name, 'builtin')
        self.variables = target.variables

Then I create a per package regression test to test each Spack-installed package. But this is for the next chapters... 😄

[sjpb]

I have a use-case for the spack load part, rather than the build system (see https://github.com/stackhpc/hpc-tests/, most of the tests there use spack-created modules). Adding support for spack load looks much easier than actually installing software using spack which IMO is a bit fragile and seems to need manual work-arounds quite a bit of the time. As @victorusu says they're orthogonal issues though.

[vkarak]

@sjpb Yes, this makes sense and is indeed easier to implement. I will open a separate issue for this one.

PS: Can I fork your repo under https://github.com/reframe-hpc ?

[vkarak]

Currently, ReFrame doesn't support to emit build commands using a job launcher, which again, I think that this is another orthogonal issue.

@victorusu I think we should bring this discussion to #1175. What we are planning there is to build remotely on the current_partition. That issue is mostly motivated from a possible lack of cross-compilation capabilities on the system that reframe runs. What you ask here is a bit broader perhaps, but it's worth putting both in the view when redesigning the build step.

[vkarak]

@sjpb @victorusu I've just opened #1584 for supporting spack load.

[sjpb]

@vkarak

PS: Can I fork your repo under https://github.com/reframe-hpc ?

Feel free - out if interest I assume this is really so its findable from ReFrame rather than because you'd intend to add PRs etc?

[vkarak]

For the moment it is just a collection of ReFrame test repositories, so that it easy to browse test examples and get ideas and other tests from other sites. In general, we are working towards facilitating the creation of generic test libraries here: https://github.com/eth-cscs/reframe/projects/23