Consolidated error handling when GPU only tests execute on CPU systems

[abouteiller]

Fixes issue #641

• capture ci tag and compile for Cuda or HIP based on this tag, not on availability of the software stack.
• decide if we keep the check in stage_main about counting GPUs. We now will detect (later) erroneous behavior and produce a parsec_fatal so maybe that is already clear enough?
• cleanup error cases for when hook==NULL, cases have been split between when the dyld function was not populated (goto next_chore) and other cases (this is a bug, assert/fail).
• stress tester is not symmetrical stress:gpu crashes when compiled with CUDA but run without a device #641 (comment)

[abouteiller]

Ok so current approach is that the tests will not fail, just put out a warning in the output.
That is not ideal but we have to choose between two bad options due to how ctest works: either we 1. cause spurious failures when running ctest when we don't have the hardware to run the test, or 2. we cause spurious success when doing so, there is no way to report that the test actually did not run.

This happens in CI, but also happens when running ctest manually (e.g., running with srun -wxyz ctest where xyz doesn't have a CUDA gpu).

@therault has been working with Geri to integrate the workflow tags into the ctest logic, but that is currently stalled so lets move with this for now.

[bosilca]

Why not returning a specific error code from the tests to indicate that the test requires hardware not available. We could then use expect to identify tests that cannot be run because the CI does not have the specific hardware (aka. the tag).

[abouteiller]

In the end, ctest will report the test as failed of passed, even if the test itself has a ternary output, ctest can only report binary results. So there will be loss of information, we just have to decide how we want it rounded up (passed) or down (failed, current behavior).
https://cmake.org/cmake/help/v3.19/prop_test/SKIP_RETURN_CODE.html