[Test] Add integration tests to validate support for GB200.

[gmarciani]

Description of changes

Add integration tests to validate support for GB200. In particular, it verifies the automated configuration of NVIDIA IMEX.

This test creates a cluster with the necessary custom actions to configure NVIDIA IMEX and verifies the following:

1. On the compute resource supporting IMEX (q1-cr1), the IMEX nodes file is configured by the prolog,
   IMEX service is healthy and no errors are reported in IMEX's or prolog's logs.
   Also, IMEX gets reconfigured when nodes belonging to the same compute resource get replaced
2. On the compute resource not supporting IMEX (q1-cr2), the IMEX nodes file is not configured by the prolog,
   keeping the default values and IMEX is not started.

The test prints in test log the full IMEX status to facilitate troubleshooting.

Important Notes

• The test is added to the develop configuration for daily execution.
• The test uses instance type g4dn to simulate a p6e-gb200 instance. This is a reasonable approximation for the test because the focus of the test is on IMEX configuration, which can be executed on g4dn as well.
• Will remove the compute node custom action once [NVIDIA-IMEX] Add test attribute for NVIDIA-imex simulation aws-parallelcluster-cookbook#3001 will be merged. at that point the test will replace the custom action with injecting the cookbook attribute to force the IMEX configuration.
• The test makes use of a compute node prolog, following the approach recommended by NVIDIA for per-Job Deployment. The advantage of this approach is that in this way IMEX can be easily restarted locally by each compute node at the right time. If the same would be done form the head node, an additional mechanism to orchestrate the IMEX reload should have been introduced.

Limitations

1. If we want to use p6e-gb200 instance types, we need to manually change the capacity reservation in the test because we do not have a way to automate such reservation today.

Tests

• [SUCCESS] test_gb200

References

• https://docs.nvidia.com/multi-node-nvlink-systems/imex-guide/overview.html
• https://docs.nvidia.com/multi-node-nvlink-systems/imex-guide/deployment.html

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.13%. Comparing base (c8cc980) to head (950f401).
⚠️ Report is 72 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #6934      +/-   ##
===========================================
- Coverage    90.21%   90.13%   -0.08%     
===========================================
  Files          181      181              
  Lines        16213    16396     +183     
===========================================
+ Hits         14627    14779     +152     
- Misses        1586     1617      +31     

Flag	Coverage Δ
unittests	90.13% <ø> (-0.08%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[himani2411]

[Non-Blocking]When do we plan to do this? Next Phase or another iteration in the coming weeks?

[gmarciani]

I suggest to address this improvement in a follow up PR once we fully understand the implications.
So far I have not observed any race condition, even re-executing the same integ test on an existing cluster multiple timnes, but it can be because we are using only 2 nodes and not a real cuda application