collecting average performance of multiple runs

[mand35]

I want to perform performance degradation tests using ReFrame. Therefore each application need to run multiple (X) times and afterwards the average performance need to be compared with reference values.

I tried to implement the test as follow, but there seems to be a conceptual issue.

• instantiate the test class (X times)

def _get_checks(**kwargs):
    PDT_rep = 2
    ret = []
    for i in range(1,PDT_rep+1):
       ret.append(IorCheck(i, PDT_rep, 64, '4m', '8g', '/scale_akl_nobackup/filesets/nobackup', 'POSIX', **kwargs))
    return ret

• each instance add the performance value to an array

 self.perf_patterns = {}
        self.perf_PDT_values['write_%d'%count] = sn.extractsingle(
                r'^Max Write:\s+(?P<write>\S+) MiB/sec', self.stdout, 'write', float)

• the X instance calculate the mean value and uses perf_pattern to compare

        if count == PDT_rep:
            sum_r = ''
            sum_w = ''
            self.perf_patterns['PDT_read_{0}_{1}_{2}'.format(procs,t_size,b_size)] = ""
            for i in range(1,PDT_rep+1):
               #sum_r += self.perf_PDT_values['read_%d'%i]  ## this also does not work
               self.perf_patterns['PDT_read_{0}_{1}_{2}'.format(procs,t_size,b_size)] += self.perf_PDT_values['read_%d'%i]

as a result I get:

  for m in finditer(patt, filename, encoding):
 File "/scale_akl_persistent/filesets/home/schoenherrm/projects/NeSI_ReFrame/reframe/utility/sanity.py", line 533, in finditer
   with open(filename, 'rt', encoding=encoding) as fp:
TypeError: expected str, bytes or os.PathLike object, not NoneType

Could you clarify the issue or do you have a simple solution running the application multiple times and evaluating the performances?

[vkarak]

Hi @mand35, the error you are getting it seems to be caused by an implicit evaluation of a sanity expression that has happened early (perhaps in your __init__). FYI, a sanity expression (which is actually an expression whose evaluation is deferred) may be implicitly evaluated if (a) you try to iterate over it, (b) take its truthy value by calling bool() explicitly or implicitly and (c) if you try to print its result. In your code snippets I can't see any of these cases, so I'd like to see the full stack trace and the test file to check exactly the cause of this problem.

Nonetheless, even if this was working what you are trying to do wouldn't work for several reasons:

1. The self.perf_PDT_values is an instance variable of each of your tests. How do you intend to share this?
2. Sharing variables across tests is not straightforward as well, because the instances that are returned by _get_checks() are not those that are actually run by the framework. The framework deep copies those every time for each programming environment it tests. You may perhaps achieve sharing using global variables inside your test file.
3. The order that tests returned by _get_checks() will run is not guaranteed if you use the asynchronous execution policy, so the X-th test may not be actually run last.

Thus, the easiest way to achieve what you want is to wrap IOR in a shell script that would invoke it many times and use ReFrame to run this script and collect the produced data.

[vkarak]

Hi @mand35, did you finally manage to achieve your result?

[mand35]

Hi @vkarak, no actually not.
Sure, I could outsource the multiple runs into a script OR analyze single Reframe runs with an external tool, but I would prefer not to do that.
I still look forward to solve the issue. I would like to define and analyze single tests in Reframe, as well as having the single results available in the logs. Furthermore, I would like to be able to analyze over all the runs and compute the average run time.

Since you mentioned, that the framework deep copies the instances, I don't see how global variables solve the issue. Actually, I always get back to the point, where I just have deferred expressions, thus I can not compute with these values.
Do you have another idea?

[vkarak]

Hi @mand35, would multiple job steps inside the generated job script help? For example, if the generated script was sth like the following, would that be acceptable for you?

#SBATCH -N 4
...

srun my_bench 100
srun my_bench 200
srun my_bench 300
...

You could then, in a single ReFrame test, analyse the aggregate output as you wish. Though not completely straightforward, it is possible to achieve it currently and in a portable way, i.e., not to hardcode the multiple sruns. Your original request would require to support dependencies among different tests, which is in our roadmap, but not yet implemented.

Could you share a stand-alone stripped down version of your tests that demonstrate the problem you're getting, so that I can have a better look?

[mand35]

Hi @vkarak, yes multiple generated job steps in the generated job script would help. We could then search for all instances of the performance objective (which can be done by using sanity.extractall).
We would loose the single values, which would be nice to store, but this could be acceptable.

Nevertheless, the idea also is: once defining complex benchmarks with a complex setup, use this as the usual BM case and additionally simply create a multi-run on top of that.
Below a simple example, I (in a first place) thought could work. It seems that this does not work due to deferred values.

Is there an easy solution I can define multiple job steps in a single ReFrame test? e.g. the (from you) mentioned multiple srun?

import os
import reframe.utility.sanity as sn

from reframe.core.pipeline import RegressionTest, RunOnlyRegressionTest

PDT_rep = 3

perf_PDT_values = ["-12345"] * PDT_rep
# Base class for Slurm simple binary tests
class HostnameCheck(RunOnlyRegressionTest):
    if all(item != "-12345" for item in perf_PDT_values):
          print("got all the values ", *perf_PDT_values)

    def __init__(self, count, PDT_rep, **kwargs):
        super().__init__('hostname_check',os.path.dirname(__file__), **kwargs)

        self.valid_systems = ['kupe:compute','kupe:login']
        self.valid_prog_environs = ['PrgEnv-cray']

        self.num_tasks = 1
        self.pre_run.append('beg_secs=$(date +%s%3N)')
        self.post_run.append('end_secs=$(date +%s%3N)')
        self.post_run.append('wallsecs=$(echo "scale=3; ($end_secs-$beg_secs)/1000" | bc -l); echo "Time taken in seconds is " $wallsecs')
        p_name = "perf_{}".format(self.num_tasks)
        self.perf_patterns = {
            p_name: sn.extractsingle(r'^Time taken in seconds is\s+(?P<'+p_name+'>\S+)',
                                     self.stdout, p_name, float)
        }

        perf_PDT_values[count] = sn.extractsingle(r'^Time taken in seconds is\s+(?P<'+p_name+'>\S+)', self.stdout, p_name, float)

        self.reference = {
            'kupe:compute': {
                'perf_1':  (100, None, 0.10),
                'PDT':  (0.1, None, 0.10)
            },
            'kupe:login': {
                'perf_1':  (100, None, 0.10),
                'PDT':  (0.001, None, 0.10)
            },
        }

        self.executable = '/bin/hostname'
        self.hostname_string = {
            'kupe:compute': r'nid\d{5}\b',
            'kupe:login': r'kupe\d{2}\b',
        }

    def setup(self, partition, environ, **job_opts):
        super().setup(partition, environ, **job_opts)
        num_matches = sn.count(sn.findall(
            self.hostname_string[partition.fullname], self.stdout))
        self.sanity_patterns = sn.assert_eq(self.num_tasks, num_matches)

def _get_checks(**kwargs):
    ret = []
    for i in range(1,PDT_rep+1):
       ret.append(HostnameCheck(i-1, PDT_rep, **kwargs))

    print("we got ", perf_PDT_values)
    return ret

[vkarak]

Hi @mand35,

Here is the code snippet that would allow you to use multiple sruns inside the generated script:

    def setup(self, partition, environ, **job_opts):
        super().setup(partition, environ, **job_opts)
        launch_cmd = ' '.join(self.job.launcher.command(self.job))
        self.job._pre_run += ['%s cat /proc/cpuinfo' % launch_cmd,
                              '%s nvidia-smi' % launch_cmd]

The use of _pre_run is a bit hacky for the moment, but there is no other way. At some point recently, we have moved the pre_run/post_run to the RegressionTest, hiding their counterparts inside the job instance and we "lost" this functionality. I've scheduled this to be fixed in the upcoming release.

Now, regarding your test, I was able to run it successfully (i.e., without any errors) both with 2.12 and the current master (2.13-dev0). The only change I had to make (apart from adding my system) was to create unique test names. From 2.12 onward, we don't allow loading tests with the same name, because this creates problems with the asynchronous execution policy. Which version were you using?

Here is my change:

    def __init__(self, count, PDT_rep, **kwargs):
        super().__init__('hostname_check_%s' % count, os.path.dirname(__file__), **kwargs)

Btw, in the upcoming release, there will be no need to use this boilerplate code or the _get_checks() method to define your tests.

[mand35]

Hi @vkarak,

thanks for the hints.
I don't get it completely, what is the difference between _pre_run and pre_run?
The pre_run idea is nice, I used it already a couple of times, but never with a parallel code.
Unfortunately, in my case I don't get anything back from self.job.launcher.command(self.job)
In the created job script I only see at that position [].

[vkarak]

I don't get it completely, what is the difference between _pre_run and pre_run?

The job._pre_run is just a copy of the self.pre_run in your test. This is a private field and, generally, you should not use it... I've just proposed it as a workaround until we add proper support for multiple parallel launches.

Unfortunately, in my case I don't get anything back from self.job.launcher.command(self.job)
In the created job script I only see at that position [].

If you are running on your login nodes or with --force-local, that's expected. Are you sure that it happens also when your scheduler is SLURM?

[vkarak]

Hi @mand35, I've just merged PR #333 that eliminates the need for the hack using the _pre_run and _post_run attributes. You can create multiple steps as follows now:

    def setup(self, partition, environ, **job_opts):
        super().setup(partition, environ, **job_opts)
        launch_cmd = ' '.join(self.job.launcher.command(self.job))
        self.pre_run += ['%s cat /proc/cpuinfo' % launch_cmd,
                         '%s nvidia-smi' % launch_cmd]

This fix is now on master and it will be available in the upcoming release.

[vkarak]

@mand35 I'm closing this, since it's practically fixed in 2.13.