Perflogs don't use test short name for parameterized tests

[casparvl]

I think the docs are slightly misaligned with the new functionality introduced in #2882

Before that changed, naming of performance log was <name_of_test_class>_<test_hash>.log. In other words: the short name was used to name the log file. This is also suggested by the example in the docs (even if it is not explicitely stated that this is the filename format).

Since the change, for parameterized tests, the performance log file is now just <name_of_test_class>.log, as all performance information for different parameters are combined in the same logfile.

I think the name of the log file (and change therein) could be better reflected in the docs. The first time I saw the test hash being omitted, I though it was a bug, but... it's a feature :). The documentation didn't clear it up for me, and also doesn't list the "Changed in ReFrame XYZ" that you usually include (which is extremely useful btw :)).

On a side note: changing this behavior could have breaking effects for people that use visualization tools to visualize there performance results, as these typically make assumptions on the structure of the logs used as input (e.g. everything in one file is one unique test, with parameter combinations being found in <test_name>_<any_hash>.log). It could have been nice to make the combination of logs for different test hashes optional (opt-in or opt-out, either would work to not have to adapt downstream visualization) to provide more stability for people developing or using downstream visualization tools.

Additionally there is another reason why people might prefer the previous structure. That structure (one log file per unique test) made it very easy to visually see if performance was stable over time: all entries in the log file should report the same performance. With the new logging structure, that's not so easy, as running tests with different parameters might be expected to have different performance (e.g. CPU vs GPU based runs).

[vkarak]

The docs are indeed buggy and we should be fix them. Thanks for spotting this!

I see your point about breaking behaviour and, yes, this change can break post-processing assumptions. I think though that's an improvement from all aspects:

1. You can now use perflogs for generating scaling plots, which was impossible before.
2. Perflogs now resemble very much to the rest of supported data sinks, such as Elastic, where we send all the test data as a single stream and users then use queries to filter in the data that they want.
3. This change will eventually make post-processing scripts more readable. Until now you would have to hardcode the mapping between the parameters and the hash code, which can be very cryptic, whereas now users can have scripts that are more explicit and more readable, e.g., "give me all perf. results where num_gpus=2 and modules=X. In fact, even with the previous perflogs you would have to do that sort of filtering for programming environments and system partitions at least. So, now, the post-processing becomes also more consistent imho.