RuntimeError: NCCL Error 3: internal error

[smallmocha]

[0] NCCL INFO cudaDriverVersion 11040
[0] NCCL INFO Bootstrap : Using eth0:10.84.253.70<0>
[0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation

File "/usr/local/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/transformers/generation/utils.py", line 1496, in generate
**model_kwargs,
File "/usr/local/lib/python3.7/site-packages/transformers/generation/utils.py", line 2528, in sample
output_hidden_states=output_hidden_states,
File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.7/site-packages/tensor_parallel/pretrained_model.py", line 76, in forward
return self.wrapped_model(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.7/site-packages/tensor_parallel/tensor_parallel.py", line 147, in forward
self.sharding_manager.synchronize_weights(self.all_cuda)
File "/usr/local/lib/python3.7/site-packages/tensor_parallel/sharding.py", line 77, in synchronize_weights
gathered_shards = all_gather(list(self.flat_shards), all_cuda=all_cuda)
File "/usr/local/lib/python3.7/site-packages/tensor_parallel/cross_device_ops.py", line 58, in all_gather
return NCCLAllGatherFunction.apply(*tensors)
File "/usr/local/lib/python3.7/site-packages/tensor_parallel/cross_device_ops.py", line 100, in forward
nccl.all_gather(inputs, outputs)
File "/usr/local/lib/python3.7/site-packages/torch/cuda/nccl.py", line 104, in all_gather
torch._C._nccl_all_gather(inputs, outputs, streams, comms)
RuntimeError: NCCL Error 3: internal error

[0] NCCL INFO Failed to open libibverbs.so[.1]
[0] NCCL INFO NET/Socket : Using [0]eth0:10.84.253.70<0>
[0] NCCL INFO Using network Socket
[1] NCCL INFO Using network Socket

[0] misc/nvmlwrap.cc:63 NCCL WARN Failed to open libnvidia-ml.so.1
[0] NCCL INFO misc/nvmlwrap.cc:179 -> 2
[1] NCCL INFO misc/nvmlwrap.cc:179 -> 2

[0] graph/xml.cc:634 NCCL WARN No NVML device handle. Skipping nvlink detection.

[1] graph/xml.cc:634 NCCL WARN No NVML device handle. Skipping nvlink detection.
[0] NCCL INFO misc/nvmlwrap.cc:179 -> 2
[1] NCCL INFO misc/nvmlwrap.cc:179 -> 2

[0] graph/xml.cc:634 NCCL WARN No NVML device handle. Skipping nvlink detection.

[1] graph/xml.cc:634 NCCL WARN No NVML device handle. Skipping nvlink detection.
[0] NCCL INFO graph/paths.cc:308 -> 2
[0] NCCL INFO graph/paths.cc:532 -> 2
[0] NCCL INFO init.cc:535 -> 2
[0] NCCL INFO init.cc:1089 -> 2
[0] NCCL INFO group.cc:64 -> 2 [Async thread]
[1] NCCL INFO graph/paths.cc:308 -> 2
[1] NCCL INFO graph/paths.cc:532 -> 2
[1] NCCL INFO init.cc:535 -> 2
[1] NCCL INFO init.cc:1089 -> 2
[1] NCCL INFO group.cc:64 -> 2 [Async thread]
[0] NCCL INFO group.cc:421 -> 3
[0] NCCL INFO group.cc:106 -> 3
[0] NCCL INFO init.cc:1226 -> 3

[GeneZC]

Also faced with this issue, any updates?