[release-3.7] [integ-tests] Use a fixed time MPI job to test job cancellation of MPI processes

[EddyMM]

Description of changes

• One of the test assertions (_test_mpi_job_termination) checks if cancellation of an MPI process does not leave any stray processes
• The MPI job executed is an All-to-All MPI job. Timing when the job is running in order to cancel it is hard to predict.
• These code changes use a simple sleep process instead with a deterministic runtime (5 mins). This way the test's check intervals have a higher guarantee of catching the MPI process while it's running.
• The test also separates the clean up step (removing the shared file) from the retry steps

Tests

• Ran the test (test_slurm::test_slurm) with Ubuntu2204

References

• [release-3.7] Reduce wait-time while checking mpi jobs #5654

Checklist

• Make sure you are pointing to the right branch.
• If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
• Check all commits' messages are clear, describing what and why vs how.
• Make sure to have added unit tests or integration tests to cover the new/modified code.
• Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

Merging #5665 (0f1ea5a) into release-3.7 (d7c44f5) will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           release-3.7    #5665   +/-   ##
============================================
  Coverage        89.89%   89.89%           
============================================
  Files              179      179           
  Lines            15366    15366           
============================================
  Hits             13813    13813           
  Misses            1553     1553           

Flag	Coverage Δ
unittests	89.89% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[NSsirena]

If we do this we are not using MPI at all because each node will launch a process that will sleep for a fixed amount of time, but no interaction will happen between the nodes.

Let's sync up about this.

[EddyMM]

The test involves checking if cancellation of an MPI process does not leave any stray processes. The message exchange may not be needed but as discussed, this test "implicitly" covers IntelMPI tests so this PR has been opened pending follow up on reviewing the timing issue on Ubuntu2204: #5666

[EddyMM]

Closed in preference of: #5665