openib thread fixes #828
Merged
openib thread fixes #828
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When using eager RDMA in debug builds the openib btl generates a sequence number for each send. The code independently updated the head index and the sequence number for the eager rdma transaction. If multiple threads enter this code at the same time and run in the following order: thread 1: update sequence (0 -> 1) thread 2: update sequence (1 -> 2) thread 2: update head (0 -> 1) thread 1: update head (1 -> 2) the sequence number for head[0] gets 1 and the sequence number for head[1] gets 0. The fix is to generate the sequence number from the head index. Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
There were several issues preventing the openib btl from running in thread multiple mode: - Missing locks in UDCM when generating a loopback endpoint. Fixed in open-mpi/ompi@8205d79. - Incorrect sequence numbers generated in debug mode. This did not prevent the openib btl from running but instead produced incorrect error messages in debug builds. - Recursive locking of the rcache lock caused by the malloc hooks. This is fixed by open-mpi#827 Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Member
Author
Member
|
@hjelmn -
/var/lib/jenkins/jobs/gh-ompi-master-pr/workspace/ompi_install1/bin/mpirun -np 2 -bind-to core -mca btl_openib_if_include mlx5_0:1 -mca pml ob1 -mca btl openib /var/lib/jenkins/jobs/gh-ompi-master-pr/workspace/ompi_install1/thread_tests/thread-tests-1.1/latency_th 8
|
Member
Author
|
@miked-mellanox That one was passing for me in multiple different configurations. Did you apply the patches from #827? |
Member
Author
|
@miked-mellanox Can you enable thread tests with the openib btl with this PR? Master now has the needed code to make this work. |
Member
|
@hjelmn - fixed. |
Member
|
bot:retest |
1 similar comment
Member
|
bot:retest |
Member
|
@hjelmn - tests passed, only warn: 14:30:47 + timeout -s SIGKILL 10m /var/lib/jenkins/jobs/gh-ompi-master-pr/workspace/ompi_install1/bin/mpirun -np 2 -bind-to core -mca btl_openib_if_include mlx4_0:1 -x MXM_RDMA_PORTS=mlx4_0:1 -mca pml ob1 -mca btl self,openib /var/lib/jenkins/jobs/gh-ompi-master-pr/workspace/ompi_install1/thread_tests/thread-tests-1.1/message_rate_th 8
14:30:48 [warn] opal_libevent2022_event_base_loop: reentrant invocation. Only one event_base_loop can run on each event_base at once.
14:30:50 Thread 4, message rate 4248.267450 messages/sec
14:30:50 [warn] opal_libevent2022_event_base_loop: reentrant invocation. Only one event_base_loop can run on each event_base at once.
14:30:50 [warn] opal_libevent2022_event_base_loop: reentrant invocation. Only one event_base_loop can run on each event_base at once.
|
jsquyres
pushed a commit
to jsquyres/ompi
that referenced
this pull request
Aug 23, 2016
rmaps base help: update binding error messages
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.