-
Notifications
You must be signed in to change notification settings - Fork 773
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is it accepted use 2 ncclComm_t/2 stream in 1 process with 1 GPU #208
Comments
Yes, you can reuse a GPU in multiple communicators. |
Thanks kwen2501, Yes I create 2 communicators for 2 thread in 1 process seperatly. the communicators-1 in process1 works in thread-1 only communicate with process-2's communicators-1. But I got some error like:
So i want to know the right way to create 2 communicators in 1 process, I read the example by nccl and google it, But find nothing about this. |
I remove calling ncclComm function in second thread, just used in thread1, my program works well. |
Sorry for replying late. Do you still have problem with the two-communicator case? If you do, do you mind posting your code here? |
Thanks kwen2501 , I fix it. the error is not caused by nccComm, It's about the GPU memory. I fix it by: call cudaSetDevice before use the GPU memory in different thread. Thanks again. |
Also, (not related to the problem you were seeing), while using two communicators on one GPU is okay, using two streams for those two communicators may sometimes lead to hang. For example, the following code cannot guarantee the two all-reduce operations are scheduled in the same order on the two GPUs: |
If i do below it also may get to hang? On process 1, GPU 1: |
And i do not understand, why this situation will be hang. the stream_a and stream_b is parallel and comm_a in GPU0 always talk with comm_a in GPU1 right? So in which scenario this will be get hang. |
It may. |
The CUDA processing model makes no guarantee about the order of execution of operations issued to independent streams. See for example here: https://devtalk.nvidia.com/default/topic/940657/processing-order-with-cuda-streams-in-7-5/ |
Thanks kwen2501, It looks like I have to arrange a special order to avoid hange. |
The easiest way is to use a single stream for those two communicators. |
It's a good idea. But in my scenario it's a little harder to use one stream. |
by the way, If I can guarantee the resource that used by nccl (like thread's count in 1 block) is smaller than the GPU's, It will avoid the hang? I check the doc and find:
The default nccl will use 1 block for communication ring and the max thread used by nccl is 256. So 256 is much smaller than the GPU's thread count right? Than in this case, the hang can be avoid? |
What you said is correct but that's an if. If your application has other kernels running on the GPU or not, they will take up resources as well. Even if there are only NCCL operation, if they are launched in a loop, SM resource constraint can still occur. |
Thanks @kwen2501 it's really helpful for me. And with your help my problem been fixed. |
I want to know does the nccl support multi-ncclComm_t/multi stream in 1 process with 1 GPU?
Like I use mpi to run distribute job, every process operate 1 GPU, I want use 2 thread in every process, and every thread has a unique ncclComm_t/cudastream communicate with the other process's thread.
process1-thread1 talk with process2-thread1,(have a unique ncclcomm)
process1-thread2 with process2-thread2,
In mpi I can use MPI_Comm_split_type to get sub-comm, How can i do the same thing in nccl?
The text was updated successfully, but these errors were encountered: