btl/openib: fix rdmacm hang

[hjelmn]

This commit is an attempt to fix a hang in finalize of rdmacm. This fixes
a path where no rdmacm client is found for an endpoint.

Fixes open-mpi/ompi#1829

Signed-off-by: Nathan Hjelm hjelmn@lanl.gov

(cherry picked from commit open-mpi/ompi@960fcd2)

Signed-off-by: Nathan Hjelm hjelmn@lanl.gov

[hjelmn]

:bot🏷️bug
:bot:milestone:v2.0.0
:bot:assign: @bharatpotnuri

[ompiteam-bot]

OMPIBot error: User bharatpotnuri is not valid for issue 1251.

[hjelmn]

@bharatpotnuri Please verify the fix and reply with :+1: to mark this as reviewed.

[hjelmn]

bah, the bot pulled the +1 out of that

:bot:nolabel:reviewed

[hjelmn]

:bot:nolabel:pushed-back

[mellanox-github]

Test FAILed.
See http://bgate.mellanox.com/jenkins/job/gh-ompi-release-pr/1822/ for details.

[hjelmn]

Jenkins found a bug that looks like it has been in rdmacm for some time. It was hidden by us disabling the openib btl when thread multiple was in use. Should pass Jenkins now.

[bharatpotnuri]

commit ed2bba2 tests fine for iwarp and fixes the stall at MPI_Finalize.
Thanks @hjelmn !

[hjelmn]

👍

[mellanox-github]

Test FAILed.
See http://bgate.mellanox.com/jenkins/job/gh-ompi-release-pr/1823/ for details.

[mellanox-github]

Test FAILed.
See http://bgate.mellanox.com/jenkins/job/gh-ompi-release-pr/1824/ for details.

[ibm-ompi]

Build Failed with GNU compiler! Please review the log, and get in touch if you have questions.

[ibm-ompi]

Build Failed with XL compiler! Please review the log, and get in touch if you have questions.

[jjhursey]

I don't know why the IBM CI tests are not showing up in the list, but here are the failure logs:
https://gist.github.com/89fdbe8184b26859afcbe1d0528dfcac
https://gist.github.com/ibm-ompi/1fefe2a65545eea068ddc825ebd5561f

[hjelmn]

@jjhursey There was a typo that I corrected before the IBM results came back. Probably why they are not showing up.

[mellanox-github]

Test FAILed.
See http://bgate.mellanox.com/jenkins/job/gh-ompi-release-pr/1825/ for details.

[mellanox-github]

Test FAILed.
See http://bgate.mellanox.com/jenkins/job/gh-ompi-release-pr/1826/ for details.

[hjelmn]

Mellanox failure is fixed by #1249. Both together should make Jenkins happy.

[jsquyres]

Per discussion on the webex today, @hjelmn will separate this into two PRs:

1. the regression fix for single-threaded fix will stay here and be in v2.0.0
2. move the multi-threaded rdmacm fix into a separate PR and put it in v2.0.1

[hjelmn]

@jsquyres Removed the threading bug fix. Now only has the regression fix.

[jsquyres]

@hjelmn Thanks
@hppritcha Once CI finishes, good to go (although until Mellanox removes the multi-threaded RDMACM test, it's going to fail)

[mellanox-github]

Test FAILed.
See http://bgate.mellanox.com/jenkins/job/gh-ompi-release-pr/1831/ for details.