#SBATCH --ntasks value

[yellowhat]

Hi,
first of all thank you for this magnificent piece of work.

I would like to run tests using spack as build system and slurm as launcher.
This is my settings.py file:

site_configuration = {
    "systems": [
        {
            "name": "cluster",
            "descr": "cluster",
            "hostnames": [".*"],
            "partitions": [
                {
                    "name": "hpc",
                    "scheduler": "slurm",
                    "launcher": "srun",
                    "access": ["--exclusive", "--partition hpc"],
                    "max_jobs": 2,
                    "environs": ["gnu"],
                },
            ],
        },
    ],
    "environments": [
          {
            "name": "gnu",
            "cc": "gcc-9",
            "cxx": "g++-9",
            "ftn": "gfortran-9"
        },
    ],
    "logging": [
        {
            "level": "debug",
            "handlers": [
                {
                    "type": "stream",
                    "name": "stdout",
                    "level": "info",
                    "format": "%(message)s"
                },
                {
                    "type": "file",
                    "level": "debug",
                    "format": "[%(asctime)s] %(levelname)s: %(check_info)s: %(message)s",  
                    "append": False
                }
            ],
            "handlers_perflog": [
                {
                    "type": "filelog",
                    "prefix": "%(check_system)s/%(check_partition)s",
                    "level": "info",
                    "format": (
                        "%(check_job_completion_time)s|reframe %(version)s|"
                        "%(check_info)s|jobid=%(check_jobid)s|"
                        "%(check_perf_var)s=%(check_perf_value)s|"
                        "ref=%(check_perf_ref)s "
                        "(l=%(check_perf_lower_thres)s, "
                        "u=%(check_perf_upper_thres)s)|"
                        "%(check_perf_unit)s"
                    ),
                    "append": True
                }
            ]
        }
    ],
}

This is the test that I am trying to run:

import reframe as rfm
import reframe.utility.sanity as sn

@rfm.simple_test
class Fio(rfm.RegressionTest):
    descr = "fio benchmark"
    valid_systems = ["cluster:hpc"]
    valid_prog_environs = ["*"]
    vars_dirs = parameter(["/mnt/resource"])
    vars_rw = parameter(["read", "write"])
    vars_bs_size = parameter([("4K", "128M"), ("4M", "2G")])
    prerun_cmds = ["set -euo pipefail"]
    executable = "fio"
    executable_opts = [
        "--name=fio",
        "--directory=$VARS_DIRS",
        "--rw=$VARS_RW",
        "--blocksize=$VARS_BS",
        "--size=$VARS_BS",
        "--iodepth=64",
        "--numjobs=$VARS_NUM_JOBS",
        "--group_reporting",
        "--direct=1",
        "--time_based",
        "--runtime=10s",
    ]
    postrun_cmds = ["echo FINISHED"]
#    num_tasks = 0
    build_system = "Spack"

    @run_after("init")
    def set_description(self):
        self.descr = f"FIO check ({self.vars_dirs})"

    @run_before("compile")
    def setup_build_system(self):
        self.build_system.specs = ["fio"]
        self.build_system.install_tree = "/shared/home/vasco/spack/opt/spack"

    @run_before("run")
    def set_variables(self):
        self.variables = {
            "VARS_DIRS": str(self.vars_dirs),
            "VARS_RW": str(self.vars_rw),
            "VARS_BS": str(self.vars_bs_size[0]),
            "VARS_SIZE": str(self.vars_bs_size[1]),
            "VARS_NUM_JOBS": "$(nproc)",
        }

    @sanity_function
    def assert_FINISHED(self):
        return sn.assert_found("FINISHED", self.stdout)

    @run_before("performance")
    def set_perf_patterns(self):
        self.perf_patterns = {
            "IOPS": sn.extractsingle("IOPS=(\S+),", self.stdout, 1, int),
            "BW": sn.extractsingle("BW=(\d+.\d+)(\S+)", self.stdout, 1, float),
        }

Some questions:

1. I have set prerun_cmds = ["set -euo pipefail"] in order for the job to fail fast, is there a way to add this as settings level, ie for all tests?
2. As it is now it will build on the login node and then run using slurm:
   • is there a way to split the build and run in 2 separate jobs run by slurm so the build will happen on the same kind of node as the run?
   • keep the dependency between nodes of the same partition, for example if I run the same test with 4 parameters on 2 partition I expect slurm 10 jobs (2 builds, 8 runs) each 4 runs, on each partition, will depend on the build; so as soon as the build is finished the 4 runs can run in parallel.
3. If no num_tasks is specified, reframe will automatically add #SBATCH --ntasks=1, instead if num_tasks = 0 if will add #SBATCH --ntasks=5000.
   The second case makes sbatch fail:

$ cat rfm_Fio__tmp_read___4K____128M___job.sh
#!/bin/bash
#SBATCH --job-name="rfm_Fio__tmp_read___4K____128M___job"
#SBATCH --ntasks=5000
#SBATCH --output=rfm_Fio__tmp_read___4K____128M___job.out
#SBATCH --error=rfm_Fio__tmp_read___4K____128M___job.err
#SBATCH --exclusive
#SBATCH --partition hpc
export VARS_DIRS=/tmp
export VARS_RW=read
export VARS_BS=4K
export VARS_SIZE=128M
export VARS_NUM_JOBS=$(nproc)
. $SPACK_ROOT/share/spack/setup-env.sh
spack env activate -V -d rfm_spack_env
spack load fio
set -euo pipefail
srun fio --name=fio --directory=$VARS_DIRS --rw=$VARS_RW --blocksize=$VARS_BS --size=$VARS_BS --iodepth=64 --numjobs=$VARS_NUM_JOBS --group_reporting --direct=1 --time_based --runtime=10s
echo FINISHED
$ sbatch rfm_Fio__tmp_read___4K____128M___job.sh
sbatch: error: Batch job submission failed: Requested node configuration is not available

Is there a way to completely avoid the #SBATCH --ntasks=* line?

Thanks

[yellowhat]

I managed to split the build and run:

import reframe as rfm
import reframe.utility.sanity as sn
from reframe.utility import udeps

@rfm.simple_test
class FioBuildTest(rfm.CompileOnlyRegressionTest):
    descr = "Fio build"
    valid_systems = ["hpc:hpc"]
    valid_prog_environs = ["*"]
    build_system = "Spack"
    
    @run_before("compile")
    def setup_build_system(self):
        self.build_system.specs = ["fio"]
        self.build_system.install_tree = "~/spack/opt/spack"

    @run_before("sanity")
    def set_sanity_patterns(self):
        self.sanity_patterns = sn.assert_found("Updating", self.stdout)

@rfm.simple_test
class FioRunTest(rfm.RunOnlyRegressionTest):
    descr = "Fio run"
    valid_systems = ["hpc:hpc"]
    valid_prog_environs = ["*"]
    vars_dirs = parameter(["/tmp"])
    vars_rw = parameter(["read", "write"])
    vars_bs_size = parameter([("4K", "128M"), ("4M", "2G")])
    prerun_cmds = ["set -euo pipefail"]
    executable = "fio"
    executable_opts = [
        "--name=fio",
        "--directory=$VARS_DIRS",
        "--rw=$VARS_RW",
        "--blocksize=$VARS_BS",
        "--size=$VARS_BS",
        "--iodepth=64",
        "--numjobs=$VARS_NUM_JOBS",
        "--group_reporting",
        "--direct=1",
        "--time_based",
        "--runtime=10s",
    ]
    postrun_cmds = ["echo FINISHED"]

    @run_after("init")
    def set_description(self):
        self.descr = f"FIO check ({self.vars_dirs})"

    @run_after("init")
    def inject_dependencies(self):
        self.depends_on("FioBuildTest", udeps.fully)

    @run_before("run")
    def set_variables(self):
        self.variables = {
            "VARS_DIRS": str(self.vars_dirs),
            "VARS_RW": str(self.vars_rw),
            "VARS_BS": str(self.vars_bs_size[0]),
            "VARS_SIZE": str(self.vars_bs_size[1]),
            "VARS_NUM_JOBS": "$(nproc)",
        }

    @sanity_function
    def assert_FINISHED(self):
        return sn.assert_found("FINISHED", self.stdout)

    @run_before("performance")
    def set_perf_patterns(self):
        self.perf_patterns = {
            "IOPS": sn.extractsingle("IOPS=(\S+),", self.stdout, 1, int),
            "BW": sn.extractsingle("BW=(\d+.\d+)(\S+)", self.stdout, 1, float),
        }

But the build step is always run on the login node even if I specified valid_systems = ["hpc:hpc"]

[ekouts]

Hi @yellowhat!

But the build step is always run on the login node even if I specified valid_systems = ["hpc:hpc"]

By default the compilation is always local. Could you try setting build_locally=False and see if that fixes your problem?
https://reframe-hpc.readthedocs.io/en/stable/regression_test_api.html#reframe.core.pipeline.RegressionTest.build_locally

[yellowhat]

@ekouts thanks the suggestion:

@rfm.simple_test
class FioBuildTest(rfm.CompileOnlyRegressionTest):
    descr = "Fio build"
    valid_systems = ["hpc:hpc"]
    valid_prog_environs = ["*"]
    build_system = "Spack"
    build_locally = False
    
    @run_before("compile")
    def setup_build_system(self):
        self.build_system.specs = ["fio"]
        self.build_system.install_tree = "~/spack/opt/spack"

    @run_before("sanity")
    def set_sanity_patterns(self):
        self.sanity_patterns = sn.assert_found("Updating", self.stdout)

is building using slurm.

May I ask about --ntasks=5000 above?

Thanks

[ekouts]

Also about questions 1 and 2:

1. I have set prerun_cmds = ["set -euo pipefail"] in order for the job to fail fast, is there a way to add this as settings level, ie for all tests?

There might be a better way, but setting prepare_cmds might be good enough for you? https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#systems-.partitions-.prepare_cmds

As it is now it will build on the login node and then run using slurm... keep the dependency between nodes of the same partition, for example if I run the same test with 4 parameters on 2 partition I expect slurm 10 jobs (2 builds, 8 runs) each 4 runs, on each partition, will depend on the build; so as soon as the build is finished the 4 runs can run in parallel.

If you want to avoid rebuilding for different parameters you might want to use dependencies and split the test in RunOnlyRegressionTest and CompileOnlyRegressionTest. Have a look at this example https://reframe-hpc.readthedocs.io/en/stable/tutorial_deps.html#tutorial-3-using-dependencies-in-reframe-tests

Finally about the --ntasks question, I am not sure but I will have a look and come back to you.

[teojgo]

@yellowhat
If you set the num_tasks to 0, then ReFrame will consider this as a flexible test https://reframe-hpc.readthedocs.io/en/stable/manpage.html?highlight=flexible#flexible-node-allocation and will try allocating all the idle nodes.

[yellowhat]

@teojgo and @ekouts thanks for your answers.
The "idle" policy is a bit problematic, in this case I am trying to allocate one entire node and each partition has a different number of cores. Is there a ways to simply instruct reFrame to not pass the "#SBATCH --ntasks=*" option?

Thanks

[yellowhat]

Or better overwrite it with number of nodes to be allocated.

[teojgo]

@yellowhat so you want to allocate multiple nodes for the build? or a single node with multiple cores?

[yellowhat]

I am interested in allocating entire nodes:

• for building: it is fine 1 node;
• for running: depending on the application and the partition a different number of nodes.

Thanks

[yellowhat]

Setting num_tasks = False result in #SBATCH --ntasks=4999.

[teojgo]

@yellowhat yes, it will try to consult slurm to decide how to run. So you have to set the num_tasks to the number of nodes you want to allocate (if each node is going to have one task).

[yellowhat]

No way to disable num_tasks?

[teojgo]

It is not possible to disable it as far as I can see. How do you want your sbatch script to look like. You don't want to emit num_tasks at all?