Michael Carilli
Latest commit e4c97f3 Oct 5, 2018


fp16_optimizer.py contains FP16_Optimizer, a Python class designed to wrap an existing Pytorch optimizer and automatically enable master parameters and loss scaling in a manner transparent to the user. To use FP16_Optimizer, only two lines of one's Python model need to change.

FP16_Optimizer API documentation

Simple examples with FP16_Optimizer

Imagenet with FP16_Optimizer

word_language_model with FP16_Optimizer

fp16_util.py contains a number of utilities to manually manage master parameters and loss scaling, if the user chooses.

Manual management documentation

The Imagenet with FP16_Optimizer and word_language_model with FP16_Optimizer directories also contain main.py files that demonstrate manual management of master parameters and static loss scaling. These examples illustrate what sort of operations FP16_Optimizer is performing automatically.