Skip to content

Unexpected DAG when using find_modules with environ-mappings in flat module scheme #1960

@casparvl

Description

@casparvl

Goal

What I'd like to do is run a test for all GROMACS modules: the foss based ones should run on partition short, while the fosscuda based ones should run on partition gpu.

What I did

My ReFrame config defines partitions short and gpu. The short partition has valid environment foss, while the gpu partition has valid environment fosscuda. Then, I define a GROMACS test that uses find_modules combined with an environ_mapping, as suggested in https://reframe-hpc.readthedocs.io/en/stable/utility_functions_reference.html?highlight=find_modules()#reframe.utility.find_modules

find_modules_partial = functools.partial(find_modules, environ_mapping={
    r'.*-foss-.*': 'foss',
    r'.*-fosscuda-.*': 'fosscuda',
})
class GromacsBase(rfm.RunOnlyRegressionTest):
    module_info = parameter(find_modules_partial('GROMACS'))
        def __init__(self):
        s, e, m = self.module_info
        self.valid_systems = [s]
....
@rfm.simple_test
class GromacsNative(GromacsBase):
    def __init__(self):
        super().__init__()
        self.tags.add('native')
        s, e, m = self.module_info
        self.valid_systems = [s]
        self.valid_prog_environs = [e]
        self.modules = [m]
...

Expected result

I expected my purned DAG to look like this

Pruned test DAG
  ('GromacsNative_singlenode___example_system_short____foss____GROMACS_2020_3_foss_2020a__', 'example_system:short', 'foss') -> []
  ('GromacsNative_singlenode___example_system_gpu____fosscuda____GROMACS_2020_3_fosscuda_2020a__', 'example_system:gpu', 'fosscuda') -> []

Instead, it looks like this

Pruned test DAG
  ('GromacsNative_singlenode___example_system_short____foss____GROMACS_2020_3_foss_2020a__', 'example_system:short', 'foss') -> []
  ('GromacsNative_singlenode___example_system_short____foss____GROMACS_2020_3_fosscuda_2020a__', 'example_system:short', 'foss') -> []
  ('GromacsNative_singlenode___example_system_gpu____fosscuda____GROMACS_2020_3_fosscuda_2020a__', 'example_system:gpu', 'fosscuda') -> []

Root cause

The e in environs on this line https://github.com/eth-cscs/reframe/blob/master/reframe/utility/__init__.py#L634 is messing with me. Basically, it will test
re.match('-fosscuda-', GROMACS/2020.3-fosscuda-2020a) and 'foss' in 'fosscuda':
which is True because the name of my first environment (foss) is contained in the name of my second environment (fosscuda). Because of this, it is not only considering modules that match r'.*-foss-.*' to be valid for foss, but also modules that match r'.*-fosscuda-.*'.

Suggested solution

I'd consider this behaviour a bug and would propose to replace
if re.match(patt, m) and e in environs:
by
if re.match(patt, m) and e == environs:
Of course, there might have been a very good reason to test e in environs to begin with, but it is creating very unintuitive behaviour...

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions