Custom parameters in report files

[stefan-wolfsheimer]

ReFrame report files have a fixed schema (reframe/schemas/runreport.json).
Test depended information can be dynamically added as perfvars or tags. In many cases it is desirable to report additional information (e.g. test input parameters or output) next to the standard result fields.
In this way, the report file will become fully self-contained and can be ingested as it is into a document database. The requested feature would enable arbitrary filtering, grouping and aggregations.

For example:

"testcases": [
{ ...
  "tags": ["this", "are", "the", "custom", "tags"],    <--- already existing feature
  "parameters": { <--- arbitrary structure would be nice to have
    "input_size": 10,        
    "method": "my_method_name_as_parameter",
    "threashold": 1.3}
}
]                          

The above data-structure would support performance graphs as function of the "input_size" of this particular test while filtering by "method".
As a workaround it is possible to extract run parameters from tags, run name or descriptions but the support of a flexible parameters structure would be much easier and less error prone.

[vkarak]

Hi @stefan-wolfsheimer would it be acceptable if all test fields were dumped? In this case, the run report could become quite large.

[stefan-wolfsheimer]

Hi @vkarak ,
what would it mean to dump all test fields? Would it imply to dump all attributes of the test object?
This would also imply that potential sensitive information would be dumped.

Maybe the developer of the test could define it just as tags:

self.parameters = {"input_size": 10,        
                              "method": "my_method_name_as_parameter",
                             "threashold": 1.3}

Or, one could write out the configuration of parameterized tests by default:

class MyTest(rfm.RunOnlyRegressionTest):
    input_size = parameter([10, 20])
    method = parameter(["my_method_name_as_parameter", "just_another_method"])
    threshold = parameter([0.1, 1.3, 1.5])

This approach would work in most of our use cases.

However, it would also be useful to report non-perfvalue information, such as strings or numbers extracted from stdout after the test. Something like:

@rfm.run_after('run')
def parse_output(self):
     '''Extract and test parameters.'''
      with osext.change_dir(self.stagedir):
           self.parameters['extrainfo'] = sn.extractsingle(r'extrainfo:(?P<extrainfo>.*?);', self.stdout, "extrainfo", str, item=-1).evaluate()

I hope I could clarify the use case and proposal. Looking forward for your feedback.

KR,

Stefan

[vkarak]

I'm generally not convinced that it's a good idea to have test variables that control what is reported and how it is reported. I think reporting of a test run is not responsibility of the test's specification. That's why I am more inclined to allow (through a configuration switch) dumping all test fields instead of the proposal here. Alternatively, we could dump only the variables and parameters, i.e. anything defined with variable or parameter, and not any test field. Your request is valid, in any case, but I'm not sure what would be the best way to implement this.

[vkarak]

I think that dumping variables and parameters is they way to go here. This will dump anything from the user that this is defined as such. That could also be in the schema in a flexible way (just an arbitrary dict).

[stefan-wolfsheimer]

The new naming scheme is already very useful as we can extract the parameters from the display_name.
When the parameters are transformed into a substructure in the report file, parsing will become a bit simpler.