Undocumented handlers_perflog records

[flx42]

I might be missing something obvious here, but the handlers_perflog reference does not seem to be documenting the performance-related arguments used in Configuring ReFrame for Your Site, e.g.:

• check_perf_value
• check_perf_var
• check_perf_ref
• check_perf_{upper,lower}_thres
• check_perf_unit

Particularly, I was looking at a way to print PASS or FAIL (or just True / False) in the perflog output, but it doesn't seem to be possible, instead one has to squint at check_perf_var and check_perf_ref + check_perf_{upper,lower}_thres

[vkarak]

Hi @flx42, this is indeed a documentation bug. As for your second point, have a look at #2394. Currently, logging simply logs the performance during the "performance" pipeline stage and not the test result. At the moment of logging the performance data, the framework does not know if the test will succeed or not. That's a limitation that will be normally addressed with #2394.

[vkarak]

I did a bit of research as of why these were left undocumented and indeed there is a reason for that. Currently, ReFrame logs a new line for each performance variable. There was the intention as discussed in #1543 and #1545 to log all the performance variables at once with the %(check_perfvalues)s and therefore eliminate the need of multiple performance log calls. As this is a breaking change, it was deferred to 4.0, but we left the check_perf_* attributes undocumented on purpose in order to favour the usage of the %(check_perfvalues)s. Now, frankly, I'm not sure what will be the best behaviour. On the other hand, the check_perf_* variables are still be used by the builtin config, so technically they should still be documented.