Enhancement: Custom user fields in environments

[OliverPerks]

Enhancement request for ReFrame: add a user 'extras' field to the environments definition.

User story:
Whilst working to integrate ReFrame to Spack I have been trying to have ReFrame feed compiler data to Spack.

In Spack we have the notation compiler_name@compiler_version - e.g. gcc@10.3.0.

I would like to somehow encode this information within an environment - so that a platform can have multiple environments which represent different Spack compiler environments.

Currently we have an 'extras' field in partitions, which we can embed extra information in - I wish this to be extended to environments.

Example:

   'environments': [
        {
            'name': 'gcc7',
            'cc': 'gcc',
            'cxx': 'g++',
            'ftn': 'gfortran',
            'extras': {
                  'spack_compiler_name': 'gcc',
                  'spack_compiler_version': '7.3.0'
            }
        },
        {
            'name': 'gcc10',
            'cc': 'gcc',
            'cxx': 'g++',
            'ftn': 'gfortran',,
            'extras': {
                  'spack_compiler_name': 'gcc',
                  'spack_compiler_version': '10.3.0'
            }
        },
        {
            'name': 'acfl21',
            'cc': 'armclang',
            'cxx': 'armclang++',
            'ftn': 'armflang',,
            'extras': {
                  'spack_compiler_name': 'arm',
                  'spack_compiler_version': '21.1.0.1069'
            }
        }
    ],

That way a simple test to check if the compiler is available could look something like this:

@rfm.simple_test
class SpackEnvTest(rfm.RunOnlyRegressionTest):

    valid_prog_environs = ['*']
    valid_systems = ['*']

    executable = 'spack'
    executable_opts = ['compiler list']

    @run_after('setup')
    def update_env(self):
        if self.current_environ:
            self.spack_compiler_name = current_environ.extras['spack_compiler_name']
            self.spack_compiler_version = current_environ.extras['spack_compiler_name']

    @run_before('run')
    def replace_launcher(self):
        self.job.launcher = getlauncher('local')()

    @sanity_function
    def validate_compiler(self):
        compiler_str = '{0}@{1}'.format(self.spack_compiler_name, self.spack_compiler_version)
        return sn.assert_found(compiler_str, self.stdout, 'compiler not found: \n{0}\n  * In list:\n{1}')

We run spack compiler list and query the output for the string of the compiler we are looking for, thus ensuring that the compiler is available before Spack builds any software using the environment compiler.

[jjotero]

While this extras field is not added to the environments in the configuration file, you could probably work around this issue by expressing that logic in the test itself (see below). I can see why you'd like to have it all in the configuration, so I'm just suggesting this as a temporary workaround.

class SpackEnvBase(rfm.RunOnlyRegressionTest):
    valid_prog_environs = ['*']
    valid_systems = ['*']

    executable = 'spack'
    executable_opts = ['compiler list']

    # Required variables
    spack_compiler_name = variable(str)
    spack_compiler_version = variable(str)

    @run_before('run')
    def replace_launcher(self):
        self.job.launcher = getlauncher('local')()

    @sanity_function
    def validate_compiler(self):
        compiler_str = '{0}@{1}'.format(self.spack_compiler_name, self.spack_compiler_version)
        return sn.assert_found(compiler_str, self.stdout, 'compiler not found: \n{0}\n  * In list:\n{1}')


@rfm.simple_test
class SpackEnvTest(SpackEnvBase):
    @run_after('setup')
    def set_spack_env(self):
        if self.current_environ.name == 'gcc7':
            self.spack_compiler_name = 'gcc'
            self.spack_compiler_version = '7.3.0'
        elif self.current_environ.name == 'gcc10':
            self.spack_compiler_name = 'gcc'
            self.spack_compiler_version = '10.3.0'
        elif self.current_environ.name == 'acfl21':
            self.spack_compiler_name = 'gcc'
            self.spack_compiler_version = '21.1.0.1069'

[OliverPerks]

Yer for the minute that's what I have done. Used the name field and split on that. But it would be nice to have some more flexibility.

[vkarak]

This is practically a duplicate of #1973.