Runtime not loading correct partitions when fixtures and/or deps are used

[mahendrapaipuri]

Looks like reframe.core.runtime.runtime() not loading correct partition when either fixtures and/or dependencies are used. A minimal reproducible is as follows:

class DownloadTest(rfm.RunOnlyRegressionTest):
 
    def __init__(self):
        self.local = True
        self.sourcesdir = None
        self.executable = 'wget'
        self.executable_opts = ['https://gist.githubusercontent.com/ramsey/11072524/raw/62dace10c306381445a0110538097d5c02227f2d/hello1.c']

    @sanity_function
    def pass_sanity(self):
        return True
        
@rfm.simple_test
class BuildTest(rfm.CompileOnlyRegressionTest):
    
    # Share resource from fixture
    src_file = fixture(DownloadTest, scope='session')

    def __init__(self):
        self.valid_prog_environs = [
            'builtin',
        ]
        self.valid_systems = [
            'generic:mypartition-1', 
            'generic:mypartition-2',
            'generic:mypartition-3',
        ]
    
    @run_before('compile')
    def set_sourcedir(self):
        """Set source path based on dependencies"""
        self.sourcesdir = self.src_file.stagedir
        self.sourcepath = 'hello1.c'

    @run_before('compile')
    def set_build_system_attrs(self):
        """Set build directory and config options"""
        self.build_system = 'SingleSource'

    @sanity_function
    def pass_sanity(self):
        return True

@rfm.simple_test
class RunTest(rfm.RunOnlyRegressionTest):
    descr = 'Test to read the created file'
    valid_systems = [
        'generic:mypartition-1', 
        'generic:mypartition-2',
        'generic:mypartition-3',
    ]
    valid_prog_environs = ['builtin']
    
    @run_after('init')
    def set_dependencies(self):
        """Set dependencies of the test"""
        self.depends_on('BuildTest', udeps.by_part)

    @require_deps
    def set_executable(self, BuildTest):
        """Set executable path and executable"""
        self.sourcesdir = os.path.join(BuildTest().stagedir)
        self.executable = './BuildTest'

    @sanity_function
    def pass_sanity(self):
        return True

And reframe_config.py is defined as follows:

site_configuration = {
    'systems': [
        {
            'name': 'generic',
            'descr': 'Generic system. When no system is matched, it falls back to generic system',
            'hostnames': ['.*'],
            'modules_system': 'nomod',
            'partitions': [
                {
                    'name': 'mypartition-1',
                    'scheduler': 'local',
                    'launcher': 'local',
                    'environs': ['builtin'],
                },
                {
                    'name': 'mypartition-2',
                    'scheduler': 'local',
                    'launcher': 'local',
                    'environs': ['builtin'],
                },
                {
                    'name': 'mypartition-3',
                    'scheduler': 'local',
                    'launcher': 'local',
                    'environs': ['builtin'],
                },
            ]
        },  # end generic
    ],
    'environments': [
        {
            'name': 'builtin',
            'cc': 'gcc',
            'cxx': 'g++',
            'ftn': 'gfortran',
            'target_systems': [
                'generic:mypartition-1',
                'generic:mypartition-2',
                'generic:mypartition-3',
            ]
        },
    ],

This test is typical sequence of downloading sources, compiling them and then running. To check which config runtime is loading, I added a print statement before line as follows:

print(f"Current partition name is {self.current_partition.name}. Partition loaded by runtime is {rt.runtime().get_option(f'systems/0/partitions')}")

The output is as follows:

[ReFrame Setup]
  version:           3.10.0-dev.3+1407ae75
  command:           'reframe/bin/reframe -C reframe_config.py -c runtime-bug.py -r'
  launched by:       mahendrapaipuri@56.1.168.192.in-addr.arpa
  working directory: '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame'
  settings file:     'reframe_config.py'
  check search path: (R) '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/runtime-bug.py'
  stage directory:   '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/stage'
  output directory:  '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/output'

[==========] Running 3 check(s)
[==========] Started on Sun Dec 26 11:54:16 2021 

[----------] started processing DownloadTest~generic (Fetch sample source code of helloworld)
[ RUN      ] DownloadTest~generic on generic:mypartition-1 using builtin
Current partition name is mypartition-1. Partition loaded by runtime is [{'name': 'mypartition-1', 'scheduler': 'local', 'launcher': 'local', 'environs': ['builtin']}]
[----------] finished processing DownloadTest~generic (Fetch sample source code of helloworld)

[----------] started processing BuildTest (Compile sample code from sources)
[ RUN      ] BuildTest on generic:mypartition-1 using builtin
[      DEP ] BuildTest on generic:mypartition-1 using builtin
[ RUN      ] BuildTest on generic:mypartition-2 using builtin
[      DEP ] BuildTest on generic:mypartition-2 using builtin
[ RUN      ] BuildTest on generic:mypartition-3 using builtin
[      DEP ] BuildTest on generic:mypartition-3 using builtin
[----------] finished processing BuildTest (Compile sample code from sources)

[----------] started processing RunTest (Test to read the created file)
[ RUN      ] RunTest on generic:mypartition-1 using builtin
[      DEP ] RunTest on generic:mypartition-1 using builtin
[ RUN      ] RunTest on generic:mypartition-2 using builtin
[      DEP ] RunTest on generic:mypartition-2 using builtin
[ RUN      ] RunTest on generic:mypartition-3 using builtin
[      DEP ] RunTest on generic:mypartition-3 using builtin
[----------] finished processing RunTest (Test to read the created file)

[----------] waiting for spawned checks to finish
[       OK ] (1/7) DownloadTest~generic on generic:mypartition-1 using builtin [compile: 0.010s run: 0.598s total: 0.633s]
[       OK ] (2/7) BuildTest on generic:mypartition-2 using builtin [compile: 0.101s run: 0.196s total: 0.583s]
[       OK ] (3/7) BuildTest on generic:mypartition-3 using builtin [compile: 0.102s run: 0.096s total: 0.591s]
[       OK ] (4/7) BuildTest on generic:mypartition-1 using builtin [compile: 0.226s run: 0.362s total: 0.644s]
Current partition name is mypartition-1. Partition loaded by runtime is [{'name': 'mypartition-3', 'scheduler': 'local', 'launcher': 'local', 'environs': ['builtin']}]
Current partition name is mypartition-2. Partition loaded by runtime is [{'name': 'mypartition-3', 'scheduler': 'local', 'launcher': 'local', 'environs': ['builtin']}]
Current partition name is mypartition-3. Partition loaded by runtime is [{'name': 'mypartition-3', 'scheduler': 'local', 'launcher': 'local', 'environs': ['builtin']}]
[       OK ] (5/7) RunTest on generic:mypartition-2 using builtin [compile: 0.010s run: 0.415s total: 0.505s]
[       OK ] (6/7) RunTest on generic:mypartition-3 using builtin [compile: 0.010s run: 0.395s total: 0.512s]
[       OK ] (7/7) RunTest on generic:mypartition-1 using builtin [compile: 0.009s run: 0.502s total: 0.562s]
[----------] all spawned checks have finished

[  PASSED  ] Ran 7/7 test case(s) from 3 check(s) (0 failure(s), 0 skipped)
[==========] Finished on Sun Dec 26 11:54:18 2021 
Run report saved in '/Users/mahendrapaipuri/.reframe/reports/run-report.json'
Log file(s) saved in '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/reframe.log', '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/reframe.out'

We can see for all partitions, runtime is loading mypartition-3 which is presumably the last subconfig it loaded during initialisation phase?

In the actual runs, the issue comes again at the same line where time_limit is evaluated to None as it is not loading the correct partition.

This happens only for tests with fixtures and/or dependencies. If we rewrite the above test as single test:

@rfm.simple_test
class RunTestWithoutDeps(rfm.RegressionTest):
    descr = 'Test to read the created file'
    valid_systems = [
        'generic:mypartition-1', 
        'generic:mypartition-2',
        'generic:mypartition-3',
    ]
    valid_prog_environs = ['builtin']
    
    # Share resource from fixture
    # src_file = fixture(DownloadTest, scope='session')
    
    @run_before('compile')
    def set_sourcedir(self):
        """Set source path based on dependencies"""
        self.sourcesdir = os.path.join(os.getcwd(), 'src')
        self.sourcepath = 'hello1.c'

    @run_before('compile')
    def set_build_system_attrs(self):
        """Set build directory and config options"""
        self.build_system = 'SingleSource'

    @require_deps
    def set_executable(self, BuildTest):
        """Set executable path and executable"""
        self.executable = './RunTestWithoutDeps'

    @sanity_function
    def pass_sanity(self):
        return True

with same reframe_config.py the output is

[ReFrame Setup]
  version:           3.10.0-dev.3+1407ae75
  command:           'reframe/bin/reframe -C reframe_config.py -c runtime-bug.py -r'
  launched by:       mahendrapaipuri@56.1.168.192.in-addr.arpa
  working directory: '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame'
  settings file:     'reframe_config.py'
  check search path: (R) '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/runtime-bug.py'
  stage directory:   '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/stage'
  output directory:  '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/output'

[==========] Running 1 check(s)
[==========] Started on Sun Dec 26 11:58:31 2021 

[----------] started processing RunTestWithoutDeps (Test to read the created file)
[ RUN      ] RunTestWithoutDeps on generic:mypartition-1 using builtin
Current partition name is mypartition-1. Partition loaded by runtime is [{'name': 'mypartition-1', 'scheduler': 'local', 'launcher': 'local', 'environs': ['builtin']}]
[ RUN      ] RunTestWithoutDeps on generic:mypartition-2 using builtin
Current partition name is mypartition-2. Partition loaded by runtime is [{'name': 'mypartition-2', 'scheduler': 'local', 'launcher': 'local', 'environs': ['builtin']}]
[ RUN      ] RunTestWithoutDeps on generic:mypartition-3 using builtin
Current partition name is mypartition-3. Partition loaded by runtime is [{'name': 'mypartition-3', 'scheduler': 'local', 'launcher': 'local', 'environs': ['builtin']}]
[----------] finished processing RunTestWithoutDeps (Test to read the created file)

[----------] waiting for spawned checks to finish
[       OK ] (1/3) RunTestWithoutDeps on generic:mypartition-2 using builtin [compile: 0.213s run: 0.233s total: 0.472s]
[       OK ] (2/3) RunTestWithoutDeps on generic:mypartition-1 using builtin [compile: 0.353s run: 0.513s total: 0.893s]
[       OK ] (3/3) RunTestWithoutDeps on generic:mypartition-3 using builtin [compile: 0.113s run: 0.272s total: 0.410s]
[----------] all spawned checks have finished

[  PASSED  ] Ran 3/3 test case(s) from 1 check(s) (0 failure(s), 0 skipped)
[==========] Finished on Sun Dec 26 11:58:32 2021 
Run report saved in '/Users/mahendrapaipuri/.reframe/reports/run-report.json'
Log file(s) saved in '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/reframe.log', '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/reframe.out'

We can see here the runtime is loading correct partitions here.

[vkarak]

Hi @mahendrapaipuri, I have a quick question first. Why don't you make BuildTest a fixture of the RunTest with environment scope? The fixtures tutorial does exactly that. Meanwhile, I will try to reproduce and investigate the issue you report here.

[mahendrapaipuri]

Hello @vkarak Thanks for the reply. Yes, that is what I did at first and then I ran into cmake issues (#2301) due to fixtures name mangling in the BuildTest. That is why I changed to hybrid fixture/deps approach. I hope that PR #2355 will fix the cmake issues so that I can use fixtures for BuildTest as well.

[vkarak]

I think the problem is somehow related to how the config is read and that the partition is not loaded correctly. Apart from #2388, even if I define a default value for time_limit, it doesn't seem to get it, where it works fine with other parameters. An easy way to check the configuration value is with --show-config=systems/0/partitions/@default/time_limit where @ is followed by the partition name. This needs further investigation to understand the source of the problem.

[mahendrapaipuri]

@vkarak Unfortunately, the patch in #2414 did not fix this issue. I made the same test as I described in the issue with a print statement before this line in the patch and I get the similar results as I posted originally:

[ReFrame Setup]
  version:           3.11.0-dev.0+4ffeb08e
  command:           '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/reframe/bin/reframe -C reframe_config.py -c runtime-bug.py -r --performance-report'
  launched by:       mahendrapaipuri@56.1.168.192.in-addr.arpa
  working directory: '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame'
  settings file:     'reframe_config.py'
  check search path: (R) '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/runtime-bug.py'
  stage directory:   '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/stage'
  output directory:  '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/output'

[==========] Running 3 check(s)
[==========] Started on Sun Feb  6 16:57:38 2022 

[----------] start processing checks
[ RUN      ] DownloadTest ~generic @generic:mypartition-1+builtin
[       OK ] (1/7) DownloadTest ~generic @generic:mypartition-1+builtin [compile: 0.019s run: 0.889s total: 0.930s]
[ RUN      ] BuildTest @generic:mypartition-1+builtin
[ RUN      ] BuildTest @generic:mypartition-2+builtin
[ RUN      ] BuildTest @generic:mypartition-3+builtin
Current partition name is mypartition-1. Partition loaded by runtime is [{'name': 'mypartition-3', 'scheduler': 'local', 'launcher': 'local', 'environs': ['builtin']}]
Current partition name is mypartition-2. Partition loaded by runtime is [{'name': 'mypartition-3', 'scheduler': 'local', 'launcher': 'local', 'environs': ['builtin']}]
Current partition name is mypartition-3. Partition loaded by runtime is [{'name': 'mypartition-3', 'scheduler': 'local', 'launcher': 'local', 'environs': ['builtin']}]
[       OK ] (2/7) BuildTest @generic:mypartition-1+builtin [compile: 6.458s run: 0.803s total: 7.312s]
[       OK ] (3/7) BuildTest @generic:mypartition-2+builtin [compile: 6.486s run: 0.779s total: 7.318s]
[       OK ] (4/7) BuildTest @generic:mypartition-3+builtin [compile: 6.513s run: 0.755s total: 7.325s]
[ RUN      ] RunTest @generic:mypartition-1+builtin
[ RUN      ] RunTest @generic:mypartition-2+builtin
[ RUN      ] RunTest @generic:mypartition-3+builtin
[       OK ] (5/7) RunTest @generic:mypartition-1+builtin [compile: 0.018s run: 1.725s total: 1.857s]
[       OK ] (6/7) RunTest @generic:mypartition-2+builtin [compile: 0.018s run: 1.725s total: 1.836s]
[       OK ] (7/7) RunTest @generic:mypartition-3+builtin [compile: 0.017s run: 1.720s total: 1.816s]
[----------] all spawned checks have finished

[  PASSED  ] Ran 7/7 test case(s) from 3 check(s) (0 failure(s), 0 skipped)
[==========] Finished on Sun Feb  6 16:57:48 2022 

Run report saved in '/Users/mahendrapaipuri/.reframe/reports/run-report.json'
Log file(s) saved in '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/reframe.log', '/Users/mahendrapaipuri/Documents/IR_INRIA/Scripts/ReFrame/reframe.out'

We can see there all the three partitions loaded the config from "last" partition. Looks like the config of last partition is "sticking" when we use dependencies/fixtures.

[vkarak]

@mahendrapaipuri Thanks for checking, I'll have a deeper look into it.

[vkarak]

This is indeed a bug and it occurs only with the asynchronous execution policy. If you run your example with --exec-policy=serial, you will see that that the selected partitions are the correct ones. This comes from the following line:

https://github.com/eth-cscs/reframe/blob/be2524212d7c462ae78c5264cbe59b24de1fb261/reframe/frontend/executors/__init__.py#L584

If you run serially, the one tests runs to completion with the current subconfig, but this is not the case with the asynchronous policy. In this case all test cases as pushed for execution, so the last subconfig stays in for all test cases. I need to check whether this is indeed needed and not just an artifact.