Assistance to broader Tag releases

[caio-davi]

Hello team,

I noticed you have been updating most of the last tagged releases with the -aws suffix. I think I can assist if you need help testing it on other libfabric providers.
We have some nightly tests running on slingshot fabric, and I could add your test bed to it. This way the releases would have been tested in at least two providers.
Please, let me know if it is something you are interested!

[rajachan]

Thanks for reaching out, we absolutely want to get back to general releases. As you rightly call out, test coverage across providers is one reason why we have not done that yet and any additional testing is welcome. Are you saying you have nightly tests running with the OFI NCCL plugin and the CXI provider already? If so, are you testing with the master branch?

FWIW, we currently track unit tests, functional tests using just the plugin interfaces, NCCL-level tests using the nccl-tests benchmark suite, and application tests to gain confidence in a release. This wiki page has some more information about what gets tested with each Pull Request: https://github.com/aws/aws-ofi-nccl/wiki/Testing-and-Benchmarking. In addition, we have a more exhaustive nightly test suite that tests changes with real applications.

[caio-davi]

Actually, these nightly tests I mentioned are tests for our internal software stack, and we are using nccl-tests as part of these tests. I apologize for the bad wording.
And no, we are not testing on the master branch. Since they are sanity tests, we use version 1.6.0, as it was the latest non-specific release (although I have been using a few -aws versions for other purposes with no major issues).
Please let me know how I can help with the testing.

[caio-davi]

Hey @rajachan, just checking in...
Do you think we can move forward with this?

[rajachan]

@caio-davi Yes! One thing we can do in the near term is extending the PR presubmit checks to include your tests with the cxi provider, to ensure commits that go into the master branch build and work as expected . We can setup any Github hooks needed for your automation to see new PRs, run the tests on your systems, and report back test results. Feel free to reach out to us at al-ofi-nccl-team@amazon.com and we can hash out specifics.