Support legacy optimizer fusion as config option#75
Conversation
…ft/DeepSpeed into olruwase/legacy_optimizer_fusion
|
Tested with the latest apex 0.1.0 and pytorch 1.3.1 and 1.4. |
|
We talked offline but adding comment here too. Can we test PyTorch 1.2 with the latest version of Apex (commit 50338df62). Unfortunately Apex doesn't do any versioning so we have to refer to specific commit hashes (the code has been at version 0.1.0 since the beginning). If this commit works we can update our submodule to point to commit 50338df62 and update our docs to show we support PyTorch 1.2, 1.3, and 1.4 which would be awesome. Each PyTorch version has it's own assumptions on CUDA/CuDNN versions but we don't know what Apex itself has in terms of CUDA/CuDNN dependencies. We should be able to test PyTorch 1.2 + commit 50338df62 of Apex with our current DeepSpeed docker image (which uses CUDA 10.0). |
|
Tested with pytorch 1.2 + new apex. |
* Support legacy optimizer fusion as config option * Configure for legacy optimizer fusion * Update configuration jsons for new apex
3 participants
New apex, version 0.1.0, implements a new style of optimizer fusion while deprecating the old style. This PR provides support for both old and new fusion styles as a configuration option.