Parametrizing tests based on module discovery

[mboisson]

It would be a really great feature if ReFrame could parametrize tests based on the available modules. For example, something like this :

@rfm.parameterized_test(*utility.all_modules('gromacs',min_version,max_version))

[vkarak]

An optional callable argument that would allow custom version comparisons would be useful, I guess, too. We don't want to implement all version conventions.

[davidedelvento]

Yes, this would be fantastic!!!

For sites running lmod, this could be easily and relatively robustly implemented using spider json format

For everyone else, I guess the output of module available will need to be parsed. I've already implemented such a parser for our system., it's just 60 lines of (dense) python code. If it can be of any help, I'll be happy to un-densify it, make it clearer for other to read, improve the unit tests I have written (which are not well done) and contribute it to this project.

[vkarak]

For sites running lmod, this could be easily and relatively robustly implemented using spider json format

Exactly!

If it can be of any help, I'll be happy to un-densify it, make it clearer for other to read, improve the unit tests I have written (which are not well done) and contribute it to this project.

Contributions are always welcome ;-)

[vkarak]

This one turns out to be less straightforward than it seems. We should answer what are the modules over which we want to parametrize? The decorator will run at test loading, before even the test is filtered per partition and programming environment. If you have the same package built for arch A and B, which is exposed by loading different base modules, then for which of the two would you parametrize the test? You would end up having different tests for each programming environment (compiler/toolchain) as well as target system. @victorusu who's working on that may have more details.

[victorusu]

I started to implement this feature, but because decorators are executed before the class instantiation. I have two questions/difficulties:

1. Qhat happens if one needs to load a module before having access to the module in question? e.g. at CSCS in order to load GROMACS one needs to

module load daint-gpu # or module load daint-mc
module load GROMACS

ReFrame is aware of those daint-gpu and daint-mc modules but at the virtual partition level.

2. What happens if the test needs to preload the programming environment before accessing the module in question? e.g. if one uses hierarchical module schemes, then one might need to first load the compiler, then load the MPI implementation, etc before the module becomes available.

Can you guys give me suggestions?

[mboisson]

Can't we rely on module spider ? It tells you what you need to load before getting the module

[victorusu]

@mboisson, for point number two yes. But what about number one (assuming that you do not have a Lmod installation)?

[mboisson]

Maybe there can be a syntax to say which primary modules to try loading first ?

[vkarak]

All these are good and can be done, but the don't address the following scenario. Assume the following test:
.

@parameterized_test(*available_modules('gromacs'))
class GromacsTest(...):
    def __init__(self, modules):
        self.valid_systems = ['x:gpu', 'x:skylake', 'x:knl']
        self.valid_prog_environs = ['gnu', 'intel']
        self.modules = modules

And that we have different s/w stacks for each compiler and for each arch. If we use spider that would give all the module combinations, for example, gnu/gromacs/20.1, gnu/gromacs/20.2, intel/gromacs/20.1, cuda/gromacs/20.1, cuda/gromacs/20.2, then you will end up with a test for x:gpu that would load the intel/gromacs module. You will have therefore to set the valid_systems and valid_prog_environs based on the modules parameter, which has a bit the inverse logic, where normally you set the modules in the test based on the system and programming environments and not vice versa. But if you're fine with that, I don't have a problem.

Another complication comes for modules systems other than Lmod. In that case, as @mboisson said, you can extend the interface of available_modules to pass the primary modules first, but this parameter should be multi-dimensional to cover the different combinations. Because if I call simply available_modules('gromacs', ['intel', 'openmpi']), I would only generate the intel/opempi/gromacs versions.

The interface needs a bit of thinking still, I believe.