Performance variables must be numbers

[sjpb]

I tried putting the output of "git commit" into a perf_var so that I could link changes in performance history to changes in the system configuration, but I get this error:

FAILURE INFO for ...
<snip>
  * Failing phase: performance
<snip>
Reason: sanity error: the value extracted for performance variable 'alaska:roce-openmpi4-ucx:git_ref' is not a number: a2cccdd-dirty

There's nothing in the docs I can see that state performance variables need to be numbers.

Is there a way around this, or a better way of achieving the same thing?

[vkarak]

Hi @sjpb, this behaviour is expected and it has been reported in #1146 (where ReFrame was just crashing instead of giving a message). The comment here describes what happens and why we expect a number: #1146 (comment)

What do you want to achieve exactly? How your references look like for this "performance variable?"

[sjpb]

Ok - so then this is really just a documentation issue. I'd assumed that with a reference which looked like this 'git_ref': (None, None, None, 'n/a'),, i.e. no reference, a non-numeric perf var would be ok.

What I was trying to achieve (and this obviously isn't the right way) was to get a git commit into the performance log, linked to a test run, so I can link changes in performance to changes in configuration. Still interested in a way of doing that!

[vkarak]

I'd assumed that with a reference which looked like this 'git_ref': (None, None, None, 'n/a'),, i.e. no reference, a non-numeric perf var would be ok.

This is something we could implement easily, since it seems that people want to use the performance variables to log non-performance information. I will open a feature request for that.

What I was trying to achieve (and this obviously isn't the right way) was to get a git commit into the performance log, linked to a test run, so I can link changes in performance to changes in configuration. Still interested in a way of doing that!

Makes sense. One way you could possibly do that currently, is to try to pass the git hash as a "unit", the last element of the tuple, and make sure that what you extract as a value for this variable from the output is a number (anything). Then the git hash will be logged as the unit of that variable. I know it's hacky, but it should work.

[sjpb]

FYI my eventual solution for this was to push the info into the tag instead. It isn't really a performance variable, and actually logically it makes more sense to have this available on each line of the performance log (like the reframe version) rather than generating a new line (="observation") for it. So maybe the current perf var functionality shouldn't be changed, although using tags is a bit hacky too. I considered using the info field but that seemed less appropriate.

[vkarak]

I agree. This one needs more thinking. How did you make it log the tags? Did you add another log format specifier for tags?

[sjpb]

I just added %(check_tags)s to the log format - then (outside of reframe) I have code which parses the perflogs.

[vkarak]

I see. There is also #1068 that requests other check fields to be logged as well and I'm thinking to make this more generic, so that you could easily select any test field to log, even custom ones. How that sounds?

[sjpb]

Yes that would work. Tags actually work fine to be honest though - as reframe nicely formats them as a comma-separated string in the log they're pretty easy to handle. I guess conceptually "tags" and "things I want to log" are different though.

[vkarak]

I'm closing this, too. It'll be addressed by #1068.