Support a dry-run mode in ReFrame

[mahendrapaipuri]

This mode can simply emit job scripts without executing them.

[mahendrapaipuri]

Some initial thoughts:

• This dry-run mode can be as simple as emitting job scripts and skipping sanity and performance checks.
• If we want to include sanity and performance checks as well, one option would be to provide the reference output files with the test. These files can be named after the test names so that in the dry run mode we assign these files to self.stdout during setup phase. We will have to use regex to match test name with file name to take into account parameterised tests. If the user do not provide these reference output files, we simply skip these checks.
• I guess such an approach would work with fixtures and/or dependencies as we map test name with output file name (Am I missing something here?).

Use cases:

• Can potentially speed up writing new tests. Currently, when I write new tests for HPC platforms, I debug them on the platform per se which can be a slower process. With such a dry-run mode, we can write tests for any scheduler/launcher on local machine and bug out the "obvious" errors in the emitted job scripts.
• Once we have the reference output files, it also helps us to debug the sanity and performance checks more easily in the dry-run mode without having to run tests.

[jjotero]

Hi @mahendrapaipuri. I think this dry-run mode will simply not work for all the scenarios that are supported by ReFrame.

In the example below, the fixture MyFixture generates a resource that MyTest consumes in an arbitrary pipeline hook. The fixture resource is generated when the fixture is executed, which means that there would be no resource to consume for MyTest if this were to run in dry-run mode. So in this scenario, running this in such way would make this even more difficult to debug.

class MyFixture(rfm.RunOnlyRegressionTest):
    executable = 'echo 4 > bananas.dat'

    @sanity_function
    def pass_sanity(self):
        return True

@rfm.simple_test
class MyTest(rfm.RunOnlyRegressionTest):
    valid_systems = ['*']
    valid_prog_environs = ['*']
    executable = 'echo'

    fixt = fixture(MyFixture)

    @run_after('setup')
    def consume_bananas(self):
        with open(os.path.join(self.fixt.stagedir, 'bananas.dat'), 'r') as f:
            self.bananas = int(f.read())

    @sanity_function
    def pass_sanity(self):
        return self.bananas

As for debugging sanity and performance functions, you can do that in a python shell as shown here https://reframe-hpc.readthedocs.io/en/stable/tutorial_tips_tricks.html#debugging-sanity-and-performance-patterns

[mahendrapaipuri]

Hello @jjotero, I completely agree with you that this dry-run mode will not be able to support all sort of cases that can be possible with fixtures and dependencies. In your particular example, if we provide a reference bananas.dat file, we do not have to execute MyFixture. As we do with src/ folder, just copy the reference output file to stagedir with appropriate name.

But again, I agree that things will become complicated if fixtures are managing variables that are used in the ReFrame tests. One option would be to scope this dry run to tests decorated with @rfm.simple_test and executing all fixtures. At the same time, I do not know all the subtle details of fixtures at the moment, so may be I am completely wrong here.

Anyways, I see that fixtures are available now on the master. I will play around with them to better understand their capabilities.