Conversation
deepspeed/runtime/engine.py
Outdated
| if (self.optimizer_name() == LAMB_OPTIMIZER or self.optimizer_name() == ONEBIT_LAMB_OPTIMIZER): | ||
| assert (self.dynamic_loss_scale()), "DeepSpeed {} optimizer requires dynamic loss scaling".format( | ||
| self.optimizer_name()) | ||
| assert (self.dynamic_loss_scale() and not self.bfloat16_enabled() |
There was a problem hiding this comment.
Will try switching to check the optimizer wrapper as BFLOAT16 rather than just checking if bfloat is enabled
|
@loadams, let's add a unit test as well. |
|
@loadams Just fail on this assert when using lamb with bf16. May I ask if this will keep going? |
Hi @Liangliang-Ma - apologies, I lost track of this PR. I'll work on getting this PR updated and merged. |
@Liangliang-Ma - does this branch resolve your issue? Or do you have any other feedback on the PR? |
Yes, this one works. |
|
Failing HPU tests are a transformers issue that should be fixed in transformers soon. |
3 participants
No description provided.