We should explore using TorchDynamo to capture/optimize optimizer code. I don't anticipate any major issues here, but we should make sure it plays nice with AOT Autograd. cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
