All `CompileOnlyRegressionTest` before `RunOnlyRegressionTest`

[yellowhat]

Hi,
consider the following:

import reframe as rfm
import reframe.utility.sanity as sn
from reframe.utility import udeps
from reframe.core.decorators import run_after, run_before

@rfm.simple_test
class ABuildTest(rfm.CompileOnlyRegressionTest):
    descr = "A Build"
    valid_systems = ["hpc:local"]
    valid_prog_environs = ["*"]
    build_system = "Spack"

    @run_before("compile")
    def setup_build_system(self):
        self.build_system.specs = ["fio"]
        self.build_system.install_tree = "$SPACK_ROOT/opt/spack"

    @run_before("sanity")
    def set_sanity_patterns(self):
        self.sanity_patterns = sn.assert_found("Updating", self.stdout)


@rfm.simple_test
class ALRunTest(rfm.RunOnlyRegressionTest):
    descr = "A Run"
    valid_systems = ["hpc:hpc"]
    valid_prog_environs = ["*"]
    case = parameter([0, 1])
    executable = "date"

    @run_after("init")
    def set_depends_on(self):
        self.depends_on("ABuildTest", udeps.fully)

    @sanity_function
    def assert_passed(self):
        return True


@rfm.simple_test
class BBuildTest(rfm.CompileOnlyRegressionTest):
    descr = "B Build"
    valid_systems = ["hpc:local"]
    valid_prog_environs = ["*"]
    build_system = "Spack"

    @run_before("compile")
    def setup_build_system(self):
        self.build_system.specs = ["fio"]
        self.build_system.install_tree = "$SPACK_ROOT/opt/spack"

    @run_before("sanity")
    def set_sanity_patterns(self):
        self.sanity_patterns = sn.assert_found("Updating", self.stdout)


@rfm.simple_test
class BLRunTest(rfm.RunOnlyRegressionTest):
    descr = "B Run"
    valid_systems = ["hpc:hpc"]
    valid_prog_environs = ["*"]
    case = parameter([2, 3, 4])
    executable = "date"

    @run_after("init")
    def set_depends_on(self):
        self.depends_on("BBuildTest", udeps.fully)

    @sanity_function
    def assert_passed(self):
        return True

As you can see there are 2 applications (A and B) and each one is divided into CompileOnlyRegressionTest and RunOnlyRegressionTest.

Even if the dependencies are correclty defined:

$ ./reframe/bin/reframe --checkpath deps --list -vv
...
Full test DAG:
  ('ABuildTest', 'hpc:local', 'gcc') -> []
  ('ALRunTest_0', 'hpc:hpc', 'gcc') -> [('ABuildTest', 'hpc:local', 'gcc')]
  ('ALRunTest_1', 'hpc:hpc', 'gcc') -> [('ABuildTest', 'hpc:local', 'gcc')]
  ('BBuildTest', 'hpc:local', 'gcc') -> []
  ('BLRunTest_2', 'hpc:hpc', 'gcc') -> [('BBuildTest', 'hpc:local', 'gcc')]
  ('BLRunTest_3', 'hpc:hpc', 'gcc') -> [('BBuildTest', 'hpc:local', 'gcc')]
  ('BLRunTest_4', 'hpc:hpc', 'gcc') -> [('BBuildTest', 'hpc:local', 'gcc')]
...

On running (./reframe/bin/reframe --checkpath deps --run -v) it wil run ABuildTest first then BBuildTest and then all the ALRunTest_*/BLRunTest_* in parallel.
Is there a particular reason why reFrame is running all the CompileOnlyRegressionTest first and then RunOnlyRegressionTest?

Thanks

[vkarak]

Hi @yellowhat this is just an implementation detail and both ways, generally, should be possible and valid. I can understand why you'd like to run them in depth-first order, since the build phase is currently serial, so when BBuildTest is scheduled it blocks everything. This is a general shortcoming that will be addressed by #1617. I can't think of a quick workaround to this now, unfortunately. I will keep this open, in order for us to consider a different topological sort of the test cases, so that you don't end up having to wait for large build jobs.

[yellowhat]

Ok. Thanks. I will follow #1617.