[test] Port ReFrame checks on Arolla and Tsa with PE 19.04

[lucamar]

The pull request defines the checks that are currently run on the MCH system Kesch on the new systems Arolla and Tsa. Only few checks still need to be adapted (Fieldextra,LETKF,...).

Expected failures in this pull request that are still under investigation (checked if fixed):

• Memory Overconsumption -> https://webrt.cscs.ch/Ticket/Display.html?id=38270
• Nvprof -> https://webrt.cscs.ch/Ticket/Display.html?id=38272
• CDO -> https://jira.cscs.ch/browse/UES-710
• MCH Collectives -> https://jira.cscs.ch/browse/UES-711

[vkarak]

lgtm now. Thanks @lucamar for the effort!

[vkarak]

@lucamar The PR seems to be breaking CDO tests on Kesch: https://jenkins.cscs.ch/blue/organizations/jenkins/ReframeCI/detail/ReframeCI/2966/pipeline/20, whereas they don't fail on master.

[lucamar]

@vkarak Thanks for the information, my last commit should have fixed the issue with CDO on Kesch.

[vkarak]

Thanks @lucamar. If the CI passes except for the failures mentioned in the description, I will merge this PR.

[vkarak]

@lucamar Some more tests are broken on Kesch. Tsa/Arolla seem fine. Can you please check?

[lucamar]

I was not able to reproduce the failure of AllocSpeedTest_no on Kesch, therefore I will only test Tsa for the moment. Furthermore, Kesch post-processing nodes are currently heavily used.

[lucamar]

@jenkins-cscs retry tsa

[lucamar]

The other failure on Kesch is KernelLatencyTest_sync, but after I fixed it I still see that the check fails due to low performance:

  * Reason: performance error: failed to meet reference: latency=13.846, expected 12.0 (l=-inf, u=13.200000000000001)
------------------------------------------------------------------------------
  * Reason: performance error: failed to meet reference: latency=14.1951, expected 12.0 (l=-inf, u=13.200000000000001)
------------------------------------------------------------------------------
  * Reason: performance error: failed to meet reference: latency=13.7136, expected 12.0 (l=-inf, u=13.200000000000001)
------------------------------------------------------------------------------
  * Reason: performance error: failed to meet reference: latency=13.2608, expected 12.0 (l=-inf, u=13.200000000000001)
------------------------------------------------------------------------------
  * Reason: performance error: failed to meet reference: latency=13.8325, expected 12.0 (l=-inf, u=13.200000000000001)
------------------------------------------------------------------------------
  * Reason: performance error: failed to meet reference: latency=13.2889, expected 12.0 (l=-inf, u=13.200000000000001)
------------------------------------------------------------------------------
  * Reason: performance error: failed to meet reference: latency=13.9814, expected 12.0 (l=-inf, u=13.200000000000001)
------------------------------------------------------------------------------
  * Reason: performance error: failed to meet reference: latency=13.6442, expected 12.0 (l=-inf, u=13.200000000000001)
------------------------------------------------------------------------------

Therefore I have increased the expected value to the average of the above 8 results (i.e. 13.7).

[lucamar]

I can confirm that now only the checks expected to fail are not yet passing on Tsa (see list above).
I have pushed the fix for the KernelLatencyTest_sync on Kesch and I will monitor the results.

[lucamar]

@jenkins-cscs retry kesch

[vkarak]

Only a couple of minor comments. I could fix them.

[vkarak]

For DGEMM and GPU burn on Kesch we should open internal issues.

[vkarak]

@jenkins-cscs retry kesch tsa

[vkarak]

Thanks @lucamar for this PR. I will now merge it. Check UES-717 and UES-718 for the failures on Kesch.