Propose Out-of-tree Backend Integration (PrivateUse1) as a module and FFFrog as the maintainer

[jgong5]

Stack from ghstack (oldest at bottom):

• -> Propose Out-of-tree Backend Integration (PrivateUse1) as a module and FFFrog as the maintainer #165958

I'd like to propose a new module Out-of-tree Backend Integration via PrivateUse1 device key. The out-of-tree backend integration via PrivateUse1 device key has been a recommended mechanism of plug-in third-party accelerator devices into PyTorch. There are already quite a few documents/tutorials on the usage with the primary one as https://docs.pytorch.org/docs/main/accelerator/index.html.

We also saw more and more HW vendors to leverage the PrivateUse1 mechanism to support their accelerators. For example:

1. Ascend NPU
2. Microsoft MAIA
3. MooreThreads MUSA
4. Cambricon MLU

The scope of PrivateUse1 based out-of-tree backend integration is composed of two parts:

1. PrivateUse1 device as an out-of-tree backend that involves:
   (a) make PrivateUse1 a function-complete device as other in-tree devices: i.e., device runtime, autograd, autocast, profiling, distributed, quantization etc.
   (b) a pluggable design to allow out-of-tree integration to extend the functionality of PrivateUse1 such as a backend registration mechanism that allows user-friendly device naming, runtime extension points with either C++ and Python for third-party to plug-in their runtime implementation, customizable tensor implementation for third-party to add extra info/functionality to the tensor and their serialization.
2. OpenReg: A test suite and documentation effort to guarantee the functional correctness of PrivateUse1 mechanism and to guide HW vendors with the right implementation.

I'm also proposing @fffrog as the module maintainer for this new module due to his continuous contribution to the design and implementation both parts of the module. Below are the RFCs/Feature Proposals @fffrog was working on:

1. An improvement of PrivateUse1 mechanism, facilitating third-party backend integration
2. The interoperability Standard of Third-party Backend Integration Mechanism
3. PyTorch Backend Accelerator Integration Verification and Guidance

@fffrog contributed 240+ PRs and a majority of them is related to PrivateUse1. (https://github.com/pytorch/pytorch/pulls?q=is%3Apr+author%3Afffrog+). He also reviewed 50+ PRs related to this area. He is also the primary author of OpenReg.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/165958

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit 541f636 with merge base d7d4bb7 ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / linux-jammy-rocm-py3.10 / test (default, 2, 2, linux.rocm.gpu.gfx942.1) (gh) (disabled by #165872 but the issue was closed recently and a rebase is needed to make it pass)
  test/test_cuda.py::TestCudaMallocAsync::test_allocator_backend

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

• trunk / linux-jammy-py3-clang12-executorch / test (executorch, 1, 1, linux.2xlarge, unstable) (gh) (#166072)
  export/tests/test_target_recipes.py::TestTargetRecipes::test_w2l_model

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[fffrog]

@pytorchbot label "topic: not user facing"

[fffrog]

@albanD @jgong5 Thank you a lot for your recognition and proposal.

[albanD]

Sounds great to me. Thanks @fffrog for all your contributions!

Let's wait for other core maintainers to weigh in before merging please.

[malfet]

LGTM

[fffrog]

Thank you all very much :D @albanD @malfet @ezyang @jgong5

[ezyang]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[jgong5]

Thank you all very much :D @albanD @malfet @ezyang @jgong5

Echo @fffrog Thank you @albanD @malfet @ezyang for the approval.