[develop] [integ-tests] Use a fixed time MPI job to test job cancellation of MPI processes

[EddyMM]

• Cherry-picked from: [release-3.7] [integ-tests] Use a fixed time MPI job to test job cancellation of MPI processes #5665

Description of changes

• One of the test assertions (_test_mpi_job_termination) checks if cancellation of an MPI process does not leave any stray processes
• The MPI job executed is an All-to-All MPI job. Timing when the job is running in order to cancel it is hard to predict.
• These code changes use a simple sleep process instead with a deterministic runtime (5 mins). This way the test's check intervals have a higher guarantee of catching the MPI process while it's running.
• The test also separates the clean up step (removing the shared file) from the retry steps

Tests

• Ran the test (test_slurm::test_slurm) with Ubuntu2204

References

• [release-3.7] Reduce wait-time while checking mpi jobs #5654

Checklist

• Make sure you are pointing to the right branch.
• If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
• Check all commits' messages are clear, describing what and why vs how.
• Make sure to have added unit tests or integration tests to cover the new/modified code.
• Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

Merging #5671 (ca3a508) into develop (3056029) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #5671   +/-   ##
========================================
  Coverage    89.89%   89.89%           
========================================
  Files          179      179           
  Lines        15369    15369           
========================================
  Hits         13816    13816           
  Misses        1553     1553           

Flag	Coverage Δ
unittests	89.89% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.