cutorch.synchronize() does not work

[arashno]

I installed NCCL and I am trying to use it.
Without using NCCL, everythings seems fine but slow.
But when I am trying to use NCCL, In my code when I call cutorch.synchronize() it just stops and it does nothing without making any error.
How can I findout the root of the problem?
Thanks

[ngimel]

Do you have a small repro? A known issue with nccl is that it will hang if some other thread calls cudaFree when nccl kernels are being scheduled. You can try running your code with env var THC_CACHING_ALLOCATOR=1, and see if problem persists.

[arashno]

I am using this code, But I changed it to use nccl.
[https://github.com/soumith/imagenet-multiGPU.torch]
I tried THC_CACHING_ALLOCATOR=1 and the problem persists.

[ngimel]

No changes are necessary to this code to use nccl, it should do it automatically. Take a look at https://github.com/facebook/fb.resnet.torch/, it also uses nccl without deadlocks. If you are adding cutorch.synchronize in such a way that is is called while nccl kernels are being scheduled, nccl will deadlock, you should make sure you are not doing that.

[arashno]

It doesn't seems that it uses nccl by default.
This is the original code in util.lua
` local model_single = model

  model = nn.DataParallelTable(1)

  for i=1, nGPU do
     cutorch.setDevice(i)
     model:add(model_single:clone():cuda(), i)
  end`

I changed it to use nccl:

`local model_single = model

  model = nn.DataParallelTable(1,true,true)

  model:threads(function()
       require 'cudnn'
     end)

  for i=1, nGPU do
     cutorch.setDevice(i)
     model:add(model_single:clone():cuda(), i)
  end`

Without doing the change it uses default communication between GPUs which is very slow.
Actually, when t uses the default communication, using 4 GPUs is slower than using just 1 GPU.

[arashno]

Also, I find out that it does work for 2 GPUs but it does hang for 4 GPUs.