Fix #209: improve socket transport performance #223
Conversation
Split transfers over multiple sockets Launch multiple threads to drive sockets Detect AWS NICs and set nsockets/nthreads accordingly
|
Hey, just wanted to know what is the status of this PR? If it fixes the performance issues reported in #209, when can we expect it to be merged and a new NCCL release? |
…ocket from high-performance socket
|
Hi @myleott, I have just pushed a small commit to improve the CPU utilization of this PR. We will merge the PR into master branch in about two weeks depending on feedback. This will be part of the next NCCL release, but we do not have a concrete date set for the release yet. Please let us know if you are blocked on a release date. Thanks! |
|
HI @kwen2501, do you know when this fix will be part of the NCCL release? It seems to be pretty critical for performance. Is there any known downsides for this patch? |
|
Hi @xw285cornell, as previously mentioned, we would merge this PR into master depending on the feedback. We have some feedback for small changes, and will test it in the next few days. Once that's done, we will merge it into master. Thank you! |
|
FWIW, I observe 30% speedups on certain configurations just patching this PR. |
|
Good to hear that! Thanks for sharing! |
Also only determine nSocks/nThreads when creating the listen comm and pass it in the handle.
|
@myleott @xw285cornell This PR has been merged into master. |
|
Why is this PR still open? |
|
The manual squash we did did not reflect the PR as closed... we just need to close it manually :-) |
Split transfers over multiple sockets
Launch multiple threads to drive sockets
Detect AWS NICs and set nsockets/nthreads accordingly