[Training] DORT fails with new PyTorch changes #16355
Assignees
Labels
training
issues related to ONNX Runtime training; typically submitted using template
Comments
wschin
added
the
training
wschin
added a commit
that referenced
this issue
Jun 20, 2023
Fix #16355. The root cause change in PyTorch is [#103302](pytorch/pytorch#103302), which seem blocking calling make_fx inside a dynamo backend. Changes: 1. Move decomposition to `register_backend.py`, so we don't have to call `make_fx` inside DORT, which triggers a bunch of new exceptions. 2. Remove shape inference based on FakeTensorProp since the FX graph received from dynamo contains all shapes now. 3. Fix a macro bug so that DORT can build without CUDA. Before (3), ``` #if defined(USE_CUDA) || defined(USE_ROCM) virtual PhiloxGenerator& PhiloxGenerator__Default() = 0; #ifdef ENABLE_TRAINING_TORCH_INTEROP ... #endif #endif ``` After (3), ``` #if defined(USE_CUDA) || defined(USE_ROCM) virtual PhiloxGenerator& PhiloxGenerator__Default() = 0; #endif #ifdef ENABLE_TRAINING_TORCH_INTEROP ... #endif ``` The later one looks better since the `ENABLE_TRAINING_TORCH_INTEROP` is for Python bridge code, not for random-number-generating kernels `PhiloxGenerator`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the issue
Per #16353,
Orttraining Linux Lazy Tensor CI Pipelinefails due to PyTorch's change. Please investigate and fix DORT.To reproduce
Build from source and re-run the test.
Urgency
No response
ONNX Runtime Installation
Built from Source
ONNX Runtime Version or Commit ID
main branch
PyTorch Version
main branch
Execution Provider
Other / Unknown
Execution Provider Library Version
No response
The text was updated successfully, but these errors were encountered: