You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When we arm CUDA-IPC transport to notify an event descriptor through callback threads, today we launch work on all streams that the process has created but there maybe work only on a subset of streams. This results in process waking up almost immediately after armed if at least one stream was empty (highly likely because some peers are not even CUDA-IPC accessible but there are streams created for them and there would never be any real work issued on streams dedicated for those peer GPUs). The way to get around this issue is to maintain a refcount and issue callbacks if there are outstanding ops on the stream.
@Akshay-Venkatesh I just had a quick chat with @quasiben and I've pointed out that today we don't use CUDA streams with Dask. I have filed an issue in the past rapidsai/dask-cuda#96 to discuss how we could, but today using streams with Dask is not possible/beneficial. I do agree this is an important part of getting the most performance with CUDA, but I just wanted to point out that today we can't really get any performance benefits from this work, unfortunately.
@brminich Seems like finding out the number of times a file descriptor has been written is not trivial. As epoll_wait returns only the number of file descriptors that were ready, it looks like that multiple writes to the same file descriptor cannot be distinguished. Also, it seems like if we use an external file descriptor through worker params, we can expect number of file descriptors which are ready to be just one so we cannot see how many streams wrote to cuda_ipc file descriptor unless we peek into the internal state of cuda_ipc UCT. I don't see an easy way to test this case with gtest/uct. Do you have any suggestions on how we can test this?
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why ?
When we arm CUDA-IPC transport to notify an event descriptor through callback threads, today we launch work on all streams that the process has created but there maybe work only on a subset of streams. This results in process waking up almost immediately after armed if at least one stream was empty (highly likely because some peers are not even CUDA-IPC accessible but there are streams created for them and there would never be any real work issued on streams dedicated for those peer GPUs). The way to get around this issue is to maintain a refcount and issue callbacks if there are outstanding ops on the stream.
cc @bureddy @quasiben