github action: add mpi4py to v50x

[hppritcha]

There were multiple commits involved with getting mpi4py smoke test into github actions for main branch, so just do a not a cherry pick here.

bot:notacherrypick

[rhc54]

This is the intercomm create problem we see in main - not surprising, is it?

[hppritcha]

No its different. The changes to the comm_cid.c code on main that "broke" intercomm support - at least when using OB1 PML - haven't been ported back to the v5.0.x branch. Note the qualifier - i don't observe the intercomm problems if I use the UCX PML or the OFI MTL.

This failure is related to #12407

[rhc54]

I realize that is the issue that was filed - my question was if you expect the intercomm test to work given what we see in main. Sounds like you do expect it to work, and it does work in some circumstances - so it doesn't sound like something related to PMIx/PRRTE, or else it wouldn't work in all scenarios.

[hppritcha]

I realize that is the issue that was filed - my question was if you expect the intercomm test to work given what we see in main. Sounds like you do expect it to work, and it does work in some circumstances - so it doesn't sound like something related to PMIx/PRRTE, or else it wouldn't work in all scenarios.

correct the intercomm problem is not a pmix/prrte problem. i would not expect to see the intercomm failures on v5.0.x unless commit 23df181 is cherry-picked back to v5.0.x

[bosilca]

Just to make sure that everyone here understand, the comm_cid.c changes in question would have affected all the PML (ob1, ucx, *) in the same way, if the error was in comm_cid. As I said in another comment, it might be time to stop blaming that particular commit and try to understand what really is going on.

[wenduwan]

The failure is related to a pmix commit. See #12408

[wenduwan]

@hppritcha Thank you very much! I like this a lot.

Want to get opinion from @janjust too

[rhc54]

So does this mean that all future PRs to v5.0 will be committed with failed CI? Or are you going to start disabling tests over here to get this to pass? Just asking to determine if it is worth spending time on v5.0 bugs.

[hppritcha]

the mpi4py test harness has a straightforward way to disable a test based on the MPI (and possible version) that was used to build it. i figure we'd just open PRs against mpi4py main to - hopefully temporarily - disable specific tests as needed.