#TinyBert Training Pipeline Problems

[mexiQQ]

Hi Huawei team:

Sorry to disturb you, can you guys answer my following question?

Why did the training pipeline of TinyBert "general_distill.py" not use DDP to initialize the student model, instead of only initializing the teacher model? And why there is no synchronization of the normalization layer?

And when opening the mixed-precision, where can I find the function "backward" from "optimizer"？

thx

[zwjyyc]

Hi,
This code does not support fp16 training and DDP training. So the relevant part is redundant.
Please refer to the AutoTinyBERT code which supports both fp16 and DDP training.

[mexiQQ]

Thanks for your reply.