[WIP] model averaging benchmark #83
Conversation
| other_flt = 0.5 * (input.flat<float>() + other.flat<float>()); | ||
| std::copy(other.tensor_data().begin(), other.tensor_data().end(), | ||
| const_cast<char *>(input.tensor_data().begin())); | ||
| // FIXME: don't write to input tensor |
There was a problem hiding this comment.
@luomai @andrei3131 I think the memory bandwidth consumed by AverageAssign is 2/11 of the original implementation.
AverageAssign performs a single std::transform which operates on 2 tensors.
The original performs two std::copy, each operates on 2 tensors,
and I assume the expression other_flt = 0.5 * (input.flat<float>() + other.flat<float>()); opereates on 2 + 2 + 3 tensors in total.
There was a problem hiding this comment.
I noticed that this implementation gives a performance benefit on platypus2 on ResNet-32 when compared to the GPU version, but when run on 8 V100 with ResNet-50 it is worse than GPU averaging (using TF operators executed on GPU).
There was a problem hiding this comment.
This operator is register on CPU only, will it run on GPU?
There was a problem hiding this comment.
@andrei3131 means doing averaging using TensorFlow operators.
There was a problem hiding this comment.
@andrei3131 Could you point Guo to the model averaging (GPU) operator?
@lgarithm We are not mainly using this model averaging operator for experiments right now as we find out that using CPU for averaging cannot let ResNet-50 to converge while the averaging through GPU can.
There was a problem hiding this comment.
@luomai I see, that will be definitely better if the GPU is powerful.
But I think this commit would still help improve the CPU operator.
* cleanup model store cherrypicked from #83
* cleanup model store cherrypicked from #83
No description provided.