Error when using gloo as DDP backend

[Saltychtao]

Hello! Thank you for your great work on implementing VQ layer. When I use the VQ layer in DDP mode and use gloo as the backend as suggested in README, I got the following error:
terminate called after throwing an instance of 'gloo::EnforceNotMet' what(): [enforce fail at ../third_party/gloo/gloo/transport/tcp/pair.cc:510] op.preamble.length <= op.nbytes. 8773632 vs 8386560

Do you have any ideas on how to solve this problem?
I also tried to use nccl as the backend, however the program only hangs forever...

[lucidrains]

@Saltychtao Hey! So I recently fixed a bug with distributed training (and yet another one this morning in regards to nccl)

What version are you on? Could you retry on the latest version, and if that doesn't work, send me a script for reproducing the error?

[Saltychtao]

@lucidrains Hello, I have tried the newest version of the code, however it still hangs on when using nccl. Besides, I am currently using your vector-quantize-pytorch library within a complex seq2seq framework fairseq, so it will take me some time to isolate a working script that could reproduce the error. Anyway, I will send it to you as soon as possible.

[Saltychtao]

I am using this library in fairseq, and it turns that the all_reduce function from torch.distributed used in the library is not compatible with fairseq. When using functions from fairseq.distributed_utils, everything works well!