Obliged to set references in RegressionTest?

[MaximeVdB]

Hi -- firstly, I'd like to say thanks for developing ReFrame, a really nice tool!

Today I wanted to update from a 3.1-dev2 commit (c027bc7) to the 3.1 release, but this appears to introduce a non-backwards-compatible change (also the current master gives me this issue).

A somewhat minimal example looks like this, with this file as 'sources/example_matrix_vector_multiplication.c':

import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class Example1Test(rfm.RegressionTest):
    def __init__(self):
        self.descr = 'Simple matrix-vector multiplication example'
        self.valid_systems = ['*']
        self.time_limit = '00d00h01m00s'
        self.valid_prog_environs = ['*']
        self.sourcesdir = 'sources'
        self.sourcepath = 'example_matrix_vector_multiplication.c'
        self.build_system = 'SingleSource'
        self.build_system.cflags = ['-O2']
        self.executable_opts = ['1024', '100']
        self.sanity_patterns = sn.assert_found(
            r'time for single matrix vector multiplication', self.stdout)
        self.strict_check = False
        self.perf_patterns = {
            'time': sn.extractsingle(
                r'^time\sfor\ssingle\smatrix\svector\smultiplication\s(?P<perf>\S+)\ss$',
                self.stdout, 'perf', float),
        }

The setup/build/run/sanity steps seem to run fine, but the performance step crashes with

<path_to_reframe>/bin/reframe: unexpected error: not enough values to unpack (expected 5, got 4)
Traceback (most recent call last):
  File "<path_to_reframe>/reframe/frontend/cli.py", line 772, in main
    run_stats = runner.stats.json()
  File "<path_to_reframe>/reframe/frontend/statistics.py", line 143, in json
    val, ref, lower, upper, unit = ref
ValueError: not enough values to unpack (expected 5, got 4)

In 3.1 (or the current master), this can be solved by setting a 'reference' attribute like this:

        self.reference = {
            '<system>:<partition>': {
                'time': (0.0004, -0.1, 0.1, 's'),
            }
        }

but this was not needed in the prior version I was using. The different behavior is probably related to this commit: 9705d69. Is this a bug, or are we now "forced" to set references, even if only interested in performance logging and not in performance checking by ReFrame?

[teojgo]

Hello @MaximeVdB, thank you for reporting the issue which I can reproduce. It seems that during the creation of the json report, it is assumed that reference values are given. For the moment, you can add this to your tests if you don't want to specify a particular reference:

self.reference = {
    '*': {
        'time': (0, None, None, 's')
     }
}