How to parameterise tests depending on current partition?

[sjpb]

Hi, wondering if you have a suggestion how to do this:

I want to run a parameterised test using:

• all cores of the nodes
• a geometric series of number of nodes: 2, 4, 8, 16, ... up to the maximum available.

The reframe partition has a slurm --partition directive in the access property to define the slurm partition to use.

Inside a test I can parse output from sinfo to get the number of nodes and number of cores, and reading self.current_partition.access allows me to limit this to the right slurm partition. This allows me to calculate self.num_tasks_per_node and self.num_tasks for reframe.

However the current reframe partition is only set during the setup() phase, whereas test parameterisation is done BEFORE even the test's __init__(). Which makes total sense from a python perspective but is problematic here.

The best approach I can think of so far is to just parameterise over a number of tests, say range(6), then inside each parameterised test calculate the number of processes/nodes to set, with e.g. __init__(1) meaning a single node and __init__(6) meaning all nodes. But that is not really ideal as I won't get that nice series which allows me to compare 2,4,... node results across e.g. slurm partitions.

The only other idea I have is to somehow launch subtests, e.g. the main test runs on all nodes, then I run multiple srun commands to run on subsets of those. But that doesn't fit into reframe's model, and is obviously inefficient on resources.

Any suggestions gratefully appreciated!

[sjpb]

I have a working implementation of my first idea.

I define a ScalingTest class which has a @rfm.run_after('setup') function which looks at the child test classes attributes self.partition_fraction and self.node_fraction. These define the number of nodes to use and the number of tasks/processes/cores to use per node respectively. Both take either:

• a -ve int which specifies an absolute number of nodes/tasks, or
• a +ve float which specifies the factor to apply to the total number of nodes / physical cores per node.

It then queries the scheduler using info from the current reframe partition's access settings if provided, and sets the test's self.num_tasks and self.num_tasks_per_node appropriately.

Example of usage in a parameterised test where we want to use all physical cores of:

• 1 node
• 2 nodes
• 1/4 of the nodes in the scheduler partition
• 1/2 of the nodes ...
• all of the nodes ...

ends up looking like this:

NODE_STEPS = [-1, -2, 0.25, 0.5, 1.0]

@rfm.parameterized_test(*[[n] for n in NODE_STEPS])
class MyTest(ScalingTest):

    def __init__(self, num_nodes):
        
        self.partition_fraction = num_nodes
        self.node_fraction = 1.0 # use all physical cores
        ...

So that's not too bad actually but interested if there's any better ways.

[vkarak]

Hi @sjpb, one option that would fit more naturally to ReFrame and to what you want to achieve, would be to write a utility function that you would call to parametrize your test. At the time the test is created, you do have access to the system configuration with all the available partitions. You can then return combinations of partitions and number of tasks, with which you would parametrize your test. Here is an example (not tested):

def scaling_config():
    curr_sys = rt.runtime().system
    for part in curr_sys.partitions:
        if part.scheduler.registered_name not in ['slurm', 'squeue']:
            # ignore non Slurm schedulers
            continue

        num_nodes, num_cores_per_node = sinfo_magic(part.access)
        n = 1
        while n <= num_nodes:
            yield (part.fullname, n, num_cores_per_node)
            n *= 2


@rfm.parameterized_test(*scaling_config())
class MyScalingTest(rfm.RegressionTest):
    def __init__(self, part, num_tasks, num_tasks_per_node):
        self.valid_systems = [part]
        self.num_tasks = num_tasks
        self.num_tasks_per_node = num_tasks_per_node

We used the exact same approach in #1417 for allowing parameterization of tests based on the modules discovered. FYI, the API for the runtime, system and system partitions is described here in the documentation.

[sjpb]

Ah thank you that is much neater. I'd missed the option of working at that level instead of the current test.

[vkarak]

Hi @sjpb, I'm closing this. I've seen that you have implemented like that in your hpc-tests repo.