Skip to content

Conversation

jataylo
Copy link
Collaborator

@jataylo jataylo commented Dec 19, 2024

Currently the upstream example for AOTI usage breaks on ROCm (https://pytorch.org/tutorials/recipes/torch_export_aoti_python.html)

File "/root/upstream/torch/_dynamo/exc.py", line 317, in unimplemented
    raise Unsupported(msg, case_name=case_name)
torch._dynamo.exc.Unsupported: unsupported operator: aten.miopen_batch_norm.default (see https://docs.google.com/document/d/1GgvOe7C8_NVOMLOCwDaYV1mXXyHMXY7ExoewHqooxrs/edit#heading=h.64r4npvq0w0 for how to fix)

from user code:
   File "/root/vision/torchvision/models/resnet.py", line 285, in forward
    return self._forward_impl(x)
  File "/root/vision/torchvision/models/resnet.py", line 269, in _forward_impl
    x = self.bn1(x)

This PR adds a meta_registration for miopen_batch_norm to resolve this issue

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @hongxiayang @naromero77amd

@pytorch-bot
Copy link

pytorch-bot bot commented Dec 19, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/143569

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit c631a14 with merge base fd8b217 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added ciflow/rocm Trigger "default" config CI on ROCm module: rocm AMD GPU support for Pytorch labels Dec 19, 2024
@jataylo jataylo added ciflow/trunk Trigger trunk jobs on your pull request topic: bug fixes topic category topic: not user facing topic category labels Dec 19, 2024
@jataylo jataylo requested a review from jeffdaily December 19, 2024 11:58
@jataylo
Copy link
Collaborator Author

jataylo commented Dec 19, 2024

Looks like I need to double check logic here

_ TestFakeTensorCUDA.test_fake_crossref_backward_amp_nn_functional_batch_norm_cuda_float32 _
Traceback (most recent call last):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_utils.py", line 287, in torch_dispatch
_check_fake_real_tensors(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_utils.py", line 196, in _check_fake_real_tensors
torch._prims.utils.compare_tensor_meta(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_prims_common/init.py", line 178, in compare_tensor_meta
raise MetadataMismatchError(msg)
torch._subclasses.fake_tensor.MetadataMismatchError: Shapes torch.Size([0]) and torch.Size([2]) are not equal!

@jataylo
Copy link
Collaborator Author

jataylo commented Dec 24, 2024

Green, linting fixes.

@jataylo jataylo marked this pull request as ready for review December 24, 2024 15:53
@jataylo
Copy link
Collaborator Author

jataylo commented Dec 24, 2024

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@jataylo
Copy link
Collaborator Author

jataylo commented Dec 31, 2024

@pytorchbot cherry-pick --onto release/2.6 -c critical

pytorchbot pushed a commit that referenced this pull request Dec 31, 2024
…143569)

Currently the upstream example for AOTI usage breaks on ROCm (https://pytorch.org/tutorials/recipes/torch_export_aoti_python.html)

```
File "/root/upstream/torch/_dynamo/exc.py", line 317, in unimplemented
    raise Unsupported(msg, case_name=case_name)
torch._dynamo.exc.Unsupported: unsupported operator: aten.miopen_batch_norm.default (see https://docs.google.com/document/d/1GgvOe7C8_NVOMLOCwDaYV1mXXyHMXY7ExoewHqooxrs/edit#heading=h.64r4npvq0w0 for how to fix)

from user code:
   File "/root/vision/torchvision/models/resnet.py", line 285, in forward
    return self._forward_impl(x)
  File "/root/vision/torchvision/models/resnet.py", line 269, in _forward_impl
    x = self.bn1(x)
```

This PR adds a meta_registration for miopen_batch_norm to resolve this issue

Pull Request resolved: #143569
Approved by: https://github.com/jeffdaily

(cherry picked from commit 27b0d41)
@pytorchbot
Copy link
Collaborator

Cherry picking #143569

The cherry pick PR is at #144028 and it is recommended to link a critical cherry pick PR with an issue. The following tracker issues are updated:

Details for Dev Infra team Raised by workflow job

kit1980 pushed a commit that referenced this pull request Jan 7, 2025
…144028)

[ROCm] Add miopen_batch_norm to meta_registrations to fix AOTI issue (#143569)

Currently the upstream example for AOTI usage breaks on ROCm (https://pytorch.org/tutorials/recipes/torch_export_aoti_python.html)

```
File "/root/upstream/torch/_dynamo/exc.py", line 317, in unimplemented
    raise Unsupported(msg, case_name=case_name)
torch._dynamo.exc.Unsupported: unsupported operator: aten.miopen_batch_norm.default (see https://docs.google.com/document/d/1GgvOe7C8_NVOMLOCwDaYV1mXXyHMXY7ExoewHqooxrs/edit#heading=h.64r4npvq0w0 for how to fix)

from user code:
   File "/root/vision/torchvision/models/resnet.py", line 285, in forward
    return self._forward_impl(x)
  File "/root/vision/torchvision/models/resnet.py", line 269, in _forward_impl
    x = self.bn1(x)
```

This PR adds a meta_registration for miopen_batch_norm to resolve this issue

Pull Request resolved: #143569
Approved by: https://github.com/jeffdaily

(cherry picked from commit 27b0d41)

Co-authored-by: Jack Taylor <jack.taylor@amd.com>
dnikolaev-amd pushed a commit to ROCm/pytorch that referenced this pull request Jan 9, 2025
…ytorch#144028)

[ROCm] Add miopen_batch_norm to meta_registrations to fix AOTI issue (pytorch#143569)

Currently the upstream example for AOTI usage breaks on ROCm (https://pytorch.org/tutorials/recipes/torch_export_aoti_python.html)

```
File "/root/upstream/torch/_dynamo/exc.py", line 317, in unimplemented
    raise Unsupported(msg, case_name=case_name)
torch._dynamo.exc.Unsupported: unsupported operator: aten.miopen_batch_norm.default (see https://docs.google.com/document/d/1GgvOe7C8_NVOMLOCwDaYV1mXXyHMXY7ExoewHqooxrs/edit#heading=h.64r4npvq0w0 for how to fix)

from user code:
   File "/root/vision/torchvision/models/resnet.py", line 285, in forward
    return self._forward_impl(x)
  File "/root/vision/torchvision/models/resnet.py", line 269, in _forward_impl
    x = self.bn1(x)
```

This PR adds a meta_registration for miopen_batch_norm to resolve this issue

Pull Request resolved: pytorch#143569
Approved by: https://github.com/jeffdaily

(cherry picked from commit 27b0d41)

Co-authored-by: Jack Taylor <jack.taylor@amd.com>
dnikolaev-amd pushed a commit to ROCm/pytorch that referenced this pull request Jan 13, 2025
…ytorch#144028)

[ROCm] Add miopen_batch_norm to meta_registrations to fix AOTI issue (pytorch#143569)

Currently the upstream example for AOTI usage breaks on ROCm (https://pytorch.org/tutorials/recipes/torch_export_aoti_python.html)

```
File "/root/upstream/torch/_dynamo/exc.py", line 317, in unimplemented
    raise Unsupported(msg, case_name=case_name)
torch._dynamo.exc.Unsupported: unsupported operator: aten.miopen_batch_norm.default (see https://docs.google.com/document/d/1GgvOe7C8_NVOMLOCwDaYV1mXXyHMXY7ExoewHqooxrs/edit#heading=h.64r4npvq0w0 for how to fix)

from user code:
   File "/root/vision/torchvision/models/resnet.py", line 285, in forward
    return self._forward_impl(x)
  File "/root/vision/torchvision/models/resnet.py", line 269, in _forward_impl
    x = self.bn1(x)
```

This PR adds a meta_registration for miopen_batch_norm to resolve this issue

Pull Request resolved: pytorch#143569
Approved by: https://github.com/jeffdaily

(cherry picked from commit 27b0d41)

Co-authored-by: Jack Taylor <jack.taylor@amd.com>
(cherry picked from commit 0ca1e5b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/rocm Trigger "default" config CI on ROCm ciflow/trunk Trigger trunk jobs on your pull request Merged module: rocm AMD GPU support for Pytorch open source topic: bug fixes topic category topic: not user facing topic category

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants