FP16_Optimizer, a Python class designed to wrap an existing Pytorch optimizer and automatically enable master parameters and loss scaling in a manner transparent to the user. To use
FP16_Optimizer, only two lines of one's Python model need to change.
fp16_util.py contains a number of utilities to manually manage master parameters and loss scaling, if the user chooses.
The Imagenet with FP16_Optimizer and word_language_model with FP16_Optimizer directories also contain
main.py files that demonstrate manual management of master parameters and static loss scaling. These examples illustrate what sort of operations
FP16_Optimizer is performing automatically.