Tests do not validate first line (= column name) of output against reference?

[BenjaminRodenberg]

The first line of the reference results (= column names) seems to have no influence on the tests. A recent breaking change updated some of the column names in the output. However, the tests do not validate the the first line of the output against the reference results. I came to this conclusion based on this test, where the error (difference between output and reference) in the first line is not detected. As a reminder:

The first line of the reference results reads

systemtests/tests/TestCompose_nutils-of.Ubuntu1804/referenceOutput/Nutils/precice-Nutils-iterations.log

Line 1 in 0b61ba3

Timesteps Total_Iterations Iterations Convergence avgConvRate(0) deleted_Columns

However the format introduced in precice/precice#620 differs from this one. In #168 I started updating the column names correspondingly for a different test case.

What does this mean?

Currently our tests ignore the column names when comparing the output to the reference files. Therefore, we do not see errors in the column names. In the best case this leads to reference results that are inconsistent with our specification, if we forget to update the (e.g. via #168). This is ugly. In the worst case, columns might be totally mixed up and we do not notice it. This is bad.

[BenjaminRodenberg]

To make sure I modified the reference results in 90ff76e. I want to check, whether

1. the provided reference results are actually checked: Does the wrong number in 90ff76e#diff-82d470f43f7240d848cd499a50a69d8cR10 in lead to a failure?
2. the column names are not checked: 90ff76e#diff-d1e0c73ea70d1c0128af05f7d8a56d96R1 has wrong column names that do not match the output. Is this detected?

[BenjaminRodenberg]

See also #168 (comment)

[shkodm]

@BenjaminRueth , by design when comparing results all the non-numerical data (including column names) are stripped away with the following regex sed 's/(\|)\||\|>\|<//g; /[a-df-zA-Z]\|Version/d' ) (you can view more in lines https://github.com/precice/systemtests/blob/develop/compare_results.sh#L62 and above). So only numerical data is compared and thefore only changes in numerics will be detected.

This was done to remove all the system specific stuff, like version of the solver, various header in files, etc. I am not so sure if it was good idea thinking about it now, especially thinking about the columns issue described 🤷‍♂️

[BenjaminRodenberg]

Thanks @shkodm this explains the issue.

We probably just have to iteratively find out what data should be stripped away and what data should not and then tweak the script for comparison correspondingly:

• We do not want to compare version numbers or timestamps, since this will lead to unnecessary failure.
• We want to compare the column names due to the reasons I names above.

[BenjaminRodenberg]

We can use 10cfcc4 for testing the improved comparison script that is the goal of this issue.

• Do we get a proper error message for whitespace differences? eec21d3
• Are wrong column names detected (+proper error)?