btl/openib: fix cq resize calculation #1768
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Before dynamic add_procs the openib_btl_size_queues was called exactly once for non-dynamic jobs. Now the function is called on each new connection so the calculation was wrong. Re-wrote the function to correctly calculate the CQ size and only attempt to adjust the CQ if the requested size has changed. This fixes a bug when using the openib btl on psm2 hardware that is caused by the time needed to resize a CQ. The overhead was causing udcm to timeout and fail. Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Member
Author
|
@hppritcha This is the error I was telling you about. The CQ sizing was completely wrong and the overhead of resizing the CQ was breaking UDCM on omnipath hardware. This change was developed and tested on kit. |
Member
|
@hjelmn @hppritcha This looks to be a bug in UCX. We are tracking a few issues related to memory hooks being deregistered after the component is unloaded. Adding |
Member
|
bot:retest |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before dynamic add_procs the openib_btl_size_queues was called exactly
once for non-dynamic jobs. Now the function is called on each new
connection so the calculation was wrong. Re-wrote the function to
correctly calculate the CQ size and only attempt to adjust the CQ if
the requested size has changed. This fixes a bug when using the openib
btl on psm2 hardware that is caused by the time needed to resize a
CQ. The overhead was causing udcm to timeout and fail.
Signed-off-by: Nathan Hjelm hjelmn@lanl.gov