[ROCm] enable nvfuser

[jeffdaily]

Description

The nvfuser is enabled for ROCm.

Testing

CI label ciflow/trunk covers the newly enabled ROCm functionality as well as any CUDA regressions caused by these changes.

[facebook-github-bot]

🔗 Helpful links

• 🧪  See artifacts and rendered test results at hud.pytorch.org/pr/82498
• ✖️ Python docs build was skipped
• ✖️ C++ docs build was skipped
• ❓Need help or want to give feedback on the CI? Visit our office hours

✅ No Failures (0 Pending)

As of commit 6bbe7a7 (more details on the Dr. CI page):

Expand to see more

---

💚 💚 Looks good so far! There are no failures yet. 💚 💚

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[jeffdaily]

@davidberard98 added the ciflow/periodic label since this is where we run the slow tests for ROCm. It revealed an unexpected success.

2022-08-29T17:48:25.7390645Z   test_nvfuser_extremal_values_unique_consecutive_cuda_float16 (__main__.TestCudaFuserOpInfoCUDA) ... unexpected success (6.622s)
2022-08-29T17:48:32.3703269Z   test_nvfuser_extremal_values_unique_consecutive_cuda_float32 (__main__.TestCudaFuserOpInfoCUDA) ... unexpected success (6.631s)
2022-08-29T17:48:38.9635277Z   test_nvfuser_extremal_values_unique_consecutive_cuda_float64 (__main__.TestCudaFuserOpInfoCUDA) ... unexpected success (6.593s)
2022-08-29T17:48:54.6667431Z   test_nvfuser_extremal_values_unique_cuda_float16 (__main__.TestCudaFuserOpInfoCUDA) ... unexpected success (15.703s)
2022-08-29T17:49:10.5367634Z   test_nvfuser_extremal_values_unique_cuda_float32 (__main__.TestCudaFuserOpInfoCUDA) ... unexpected success (15.870s)
2022-08-29T17:49:26.3813820Z   test_nvfuser_extremal_values_unique_cuda_float64 (__main__.TestCudaFuserOpInfoCUDA) ... unexpected success (15.845s)

Should be fixed by 6bbe7a7.

Unskipped the few NVFuser tests you found in the logs. See 47da32a. They passed. So now just waiting on another round of CI.

[facebook-github-bot]

@davidberard98 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[jeffdaily]

@davidberard98 I hope the phabricator build is going well. All CI is green, including rocm slow.

[jjsjann123]

Looks mostly just mechanical changes. went through most files except the rocm specific runtime headers.

A nitpick on the cmake file to guard rocm header under USE_ROCM. not sure if there's similar thing we can do with build_variables.bzl

[davidberard98]

@jeffdaily looks good, let me know if you want to wait on anything before landing - otherwise I'll land later today (or feel free to land it yourself).

[ngimel]

@jjsjann123 are there more instances of nv-specific asm you guys are using? What about bfloat16/fp16 conversions that also use inline asm?
Edit: bfloat16 conversions are covered

[jeffdaily]

@pytorchbot merge

[pytorchmergebot]

@pytorchbot successfully started a merge job. Check the current status here.
The merge job was triggered without a flag. This means that your change will be merged once all checks on your PR have passed (ETA: 0-4 Hours). If this is not the intended behavior, feel free to use some of the other merge options in the wiki.
Please reach out to the PyTorch DevX Team with feedback or questions!

[github-actions]

Hey @jeffdaily.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.