[Feat]support horovod sync train

[a6802739]

• Use horovod to sync dense grad of the model parameter, for TrainableWrapper's grad, keep same with before.

[rhdong]

Hi @a6802739 , thank you for your contribution!

[Lifann]

And just for discussion, I noticed that Horovod implements lots of features and optimizations on its DistributedOptimizer like tensor fusion, grouped allreduce, adasum, gradient compression, etc.., in the latest-nth versions. Use the hvd.allreduce API may not reuse these features.

[a6802739]

Yeah, For tensor fusion, it's set by environment variable, and it's turn on by default, for most recommend situation, I think there is no need to care about grouped allreduce or gradient compression. But I think we could find better way to open reduction operation to the user. If we directly let user specify hvd.Sum or hvd.adamsum, we should let them import horovod before apply_gradients, or we could let user specify a string the reduction method like sum, we map this method to the corresponding hvd op, but if horovod add some reduction op, we should change the code to be compatible with newest horovod version.

[Lifann]

MLP layers, downstream from embedding, may also generate IndexedSlices gradient, need to notice that.

[a6802739]

Sorry, I never notice MLP layers could generate IndexedSlices gradient, could you give me an example?

[Lifann]

Sorry, I never notice MLP layers could generate IndexedSlices gradient, could you give me an example?

For example:

var = tf.Variable(...)
params = de.get_variable(...)
emb = de.embedding_lookup(...)
latent_tensor = sum_pooling(emb)
... = some_func(latent_tensor, var)

The some_layer are defined as:

def some_func(latent, var):
  mask = tf.greater_equal(var, threshold)
  pos = tf.where(mask)
  selected = tf.gather(var, pos)
  ...

The some_func code will make the gradient become IndexedSlices to var.

[a6802739]

I see, Thanks. Now I delelte the judgement isinstance(grad, ops.Tensor). So Horovod will check it's grad type and deal with it.

[Lifann]

LGTM