BUGs about amp

[wjn1996]

When I train the models with amp, we find the model cannot be Converged.

I find that there are some bugs in /easynlp/core/trainer.py, as shown in follow:

I analyze: the code in the red box means clearing grads, but when using amp, it cannot execute this code, which cause the problem.

Now I have resolved it and will commit a pull request in the latter.

We recommend using optimizer with amp in the following four settings:

• bertadam：self._optimizer.step() + self._optimizer.zero_grad()；
• bertadam+amp：self._scaler.step(self._optimizer) + self._scaler.update() + self._optimizer.zero_grad()
• adamw：torch.nn.utils.clip_grad_norm_() + self._optimizer.step() + self._lr_scheduler.step() + self._optimizer.zero_grad()
• adamw+amp：torch.nn.utils.clip_grad_norm_() + self._scaler.step(self._optimizer) self._scaler.update() + self._lr_scheduler.step() + self._optimizer.zero_grad()