The documentation of mixing `@performance_function`s and `perf_variables` is confusing

[vkarak]

The problem is that when no perf_variables are set by the test explicitly, ReFrame will use the performance_functions to generate the perf_variables. However, if the test sets the perf_variables explicitly, performance_functions have no (visible) effect. They are not used as performance variables at all. However, they are used as performance value extractors that can be assigned as values to the perf_variables dictionary. This is the equivalent of wrapping any function with the make_performance_function() and assigning to a performance variable. This behaviour is exactly displayed by stream4.py tutorial:

    @performance_function('MB/s')
    def extract_bw(self, kind='Copy'):
        if kind not in {'Copy', 'Scale', 'Add', 'Triad'}:
            raise ValueError(f'illegal value in argument kind ({kind!r})')

        return sn.extractsingle(rf'{kind}:\s+(\S+)\s+.*',
                                self.stdout, 1, float)

    @run_before('performance')
    def set_perf_variables(self):
        self.perf_variables = {
            'Copy': self.extract_bw(),
            'Scale': self.extract_bw('Scale'),
            'Add': self.extract_bw('Add'),
            'Triad': self.extract_bw('Triad'),
        }

The extract_bw() is not a performance variable but rather a data extractor. At the absence of the set_perf_variables, though, it would be a performance variable. That's confusing! And it can lead to strange results in case of inheritance. For example, imagine that a base test defines a set of performance_functions to be treated as performance variables and a subclass wants to add some more performance variables by setting the perf_variables directly, because that's more convenient and would avoid code repetition, then immediately it loses all the performance functions defined in the base class!

In my opinion, performance_function and perf_variables must be allowed at the same time and the dictionary would be simply built incrementally by all of them. With this solution, you could still override any of the performance functions of the base class by simply overriding the function. The following shows an example on how you can revert the performance function of the StreamMultiSysTest to that of the basic StreamTest of the tutorial:

@rfm.simple_test
class my_test(StreamMultiSysTest):
    def extract_bw(self):
        pass

    def set_perf_variables(self):
        pass

    @performance_function('MB/s', perf_key='Copy')
    def extract_copy_perf(self):
        return sn.extractsingle(r'Copy:\s+(\S+)\s+.*', self.stdout, 1, float)