Horovod Gradient Compression

[vineeths96]

I have a question about how the gradient compression is performed within Horovod. As mentioned in the source code here, we need to inherit the base compressor class and override compress and decompress functions.

What my doubt is that in which order the operations are performed. More specifically, is the order as shown below?

1. Calculate gradient on each worker.
2. Compress the gradients at each worker.
3. Communicate the gradients (compressed) to all workers.
4. Decompress the collection of gradients at each worker
5. Perform an all_reduce of gradients and update the parameters.

[tgaddair]

Hey @vineeths96, the order is:

1. Calculate gradient on each worker.
2. Compress the gradient on each worker.
3. Allreduce and average the compressed gradients across all workers.
4. Decompress the allreduced gradients.
5. Update the parameters.

So I think your intuition was correct, but the only difference is that the communication step is also when we perform the allreduce on the gradients (in compressed form to reduce network overhead).

In case you're worried about overflow / underflow, there is a PR in review that addresses this (#1949).

[vineeths96]

Hello @tgaddair.

Thank you for your reply.

I am interested in developing compressors such as QSGD. There are already a few implementations like here, where the compressed gradient at each worker is a tuple.

But as per Step 3 in your reply above, Horovod averages the compressed gradients. I am not able to understand how you would average the compressed gradients in this case - where we have a list tuples.

[xinyandai]

Hi @vineeths96,

In the original paper(line 4 of Algorithm 1), the compressed gradient will be broadcasted to all peers. In this case, I think you may need use allgather operator.

I have one imlementation for 8bit-QSGD with horovod here, which may be helpful.

[vineeths96]

Hello @xinyandai,

Thank you for your reply.

I just received enough time to go through your code. What I could understand is that you made three major changes in the Horovod Torch Distributed Optimizer:

1. You created compressors for each tensor in your class init.
2. You replaced the allreduce function with your allgather function.
3. In synchronize function you make sure you call the decompress functions with the proper arguments.

Correct me if I am wrong.