Add magic TORCH_MAKE_PYBIND_ENUM_FASTER macro #163527

swolchok · 2025-09-22T16:41:02Z

Stack from ghstack (oldest at bottom):

-> Add magic TORCH_MAKE_PYBIND_ENUM_FASTER macro #163527

See comment on the macro definition. In short, pybind11 3.x
added py::native_enum, and also had to add overhead for that new way
to bind enums on the critical path for calling functions that take
regular old py::enum_s as arguments (for example, __eq__).

Differential Revision: D82873169

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @ezyang @msaroufim @dcci @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @voznesenskym @penguinwu @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @jiayisunx @chenyang78 @kadeng @chauhang @amjames @Lucaskabela

See comment on the macro definition. In short, pybind11 3.x added `py::native_enum`, and also had to add overhead for that new way to bind enums on the critical path for calling functions that take regular old `py::enum_`s as arguments (for example, `__eq__`). Differential Revision: [D82873169](https://our.internmc.facebook.com/intern/diff/D82873169/) [ghstack-poisoned]

pytorch-bot · 2025-09-22T16:41:06Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/163527

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 9685224 with merge base 98c4e35 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

See comment on the macro definition. In short, pybind11 3.x added `py::native_enum`, and also had to add overhead for that new way to bind enums on the critical path for calling functions that take regular old `py::enum_`s as arguments (for example, `__eq__`). Differential Revision: [D82873169](https://our.internmc.facebook.com/intern/diff/D82873169/) ghstack-source-id: 311049024 Pull Request resolved: #163527

facebook-github-bot · 2025-09-22T16:41:13Z

@swolchok has exported this pull request. If you are a Meta employee, you can view the originating diff in D82873169.

See comment on the macro definition. In short, pybind11 3.x added `py::native_enum`, and also had to add overhead for that new way to bind enums on the critical path for calling functions that take regular old `py::enum_`s as arguments (for example, `__eq__`). Differential Revision: [D82873169](https://our.internmc.facebook.com/intern/diff/D82873169/) cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci EikanWang jgong5 wenzhe-nrv sanchitintel voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]

Pull Request resolved: #163527 See comment on the macro definition. In short, pybind11 3.x added `py::native_enum`, and also had to add overhead for that new way to bind enums on the critical path for calling functions that take regular old `py::enum_`s as arguments (for example, `__eq__`). ghstack-source-id: 311290067 @exported-using-ghexport Differential Revision: [D82873169](https://our.internmc.facebook.com/intern/diff/D82873169/)

facebook-github-bot · 2025-09-22T18:11:18Z

@swolchok has exported this pull request. If you are a Meta employee, you can view the originating diff in D82873169.

Skylion007 · 2025-09-22T18:29:15Z

Add a link to the pybind11 issue so we can remove this when it's resolved over there

swolchok · 2025-09-22T19:23:26Z

Add a link to the pybind11 issue so we can remove this when it's resolved over there

I haven't filed an issue and I don't see a particular reason to think it's structurally fixable. type_casters are keyed on the type they are casting, not on whether the pybind11 user registered the type with py::enum_ vs py::native_enum. Their clear recommendation is for new code to use py::native_enum.

Skylion007 · 2025-09-22T19:44:32Z

I maintain pybind11 so happy to discuss if there is a better way to optimize the casters or refactor it better in the long term

See comment on the macro definition. In short, pybind11 3.x added `py::native_enum`, and also had to add overhead for that new way to bind enums on the critical path for calling functions that take regular old `py::enum_`s as arguments (for example, `__eq__`). Differential Revision: [D82873169](https://our.internmc.facebook.com/intern/diff/D82873169/) cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci EikanWang jgong5 wenzhe-nrv sanchitintel voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]

Pull Request resolved: #163527 See comment on the macro definition. In short, pybind11 3.x added `py::native_enum`, and also had to add overhead for that new way to bind enums on the critical path for calling functions that take regular old `py::enum_`s as arguments (for example, `__eq__`). ghstack-source-id: 311326172 @exported-using-ghexport Differential Revision: [D82873169](https://our.internmc.facebook.com/intern/diff/D82873169/)

facebook-github-bot · 2025-09-22T20:10:12Z

@swolchok has exported this pull request. If you are a Meta employee, you can view the originating diff in D82873169.

See comment on the macro definition. In short, pybind11 3.x added `py::native_enum`, and also had to add overhead for that new way to bind enums on the critical path for calling functions that take regular old `py::enum_`s as arguments (for example, `__eq__`). Differential Revision: [D82873169](https://our.internmc.facebook.com/intern/diff/D82873169/) cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci EikanWang jgong5 wenzhe-nrv sanchitintel voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]

Pull Request resolved: #163527 See comment on the macro definition. In short, pybind11 3.x added `py::native_enum`, and also had to add overhead for that new way to bind enums on the critical path for calling functions that take regular old `py::enum_`s as arguments (for example, `__eq__`). ghstack-source-id: 311521197 @exported-using-ghexport Differential Revision: [D82873169](https://our.internmc.facebook.com/intern/diff/D82873169/)

facebook-github-bot · 2025-09-23T14:56:39Z

@swolchok has exported this pull request. If you are a Meta employee, you can view the originating diff in D82873169.

ezyang

Approving, but this will be annoyingly difficult to get people to remember to do lol

swolchok · 2025-09-25T19:43:57Z

this will be annoyingly difficult to get people to remember to do lol

I think it's straightforward to cause py::enum_ to break builds if this macro has not been used. Upstreaming something like this, together with optional support for said build breaks, is a possibility.

swolchok · 2025-09-25T19:44:32Z

will be bypassing test failures since the only new failure is actually a resource download error

swolchok · 2025-09-25T20:55:17Z

giving pull / linux-jammy-py3.13-clang12 a chance to rerun first

See comment on the macro definition. In short, pybind11 3.x added `py::native_enum`, and also had to add overhead for that new way to bind enums on the critical path for calling functions that take regular old `py::enum_`s as arguments (for example, `__eq__`). Differential Revision: [D82873169](https://our.internmc.facebook.com/intern/diff/D82873169/) cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta ezyang msaroufim dcci EikanWang jgong5 wenzhe-nrv sanchitintel voznesenskym penguinwu Guobing-Chen XiaobingSuper zhuhaozhe blzheng jiayisunx chenyang78 kadeng chauhang amjames Lucaskabela [ghstack-poisoned]

Pull Request resolved: #163527 See comment on the macro definition. In short, pybind11 3.x added `py::native_enum`, and also had to add overhead for that new way to bind enums on the critical path for calling functions that take regular old `py::enum_`s as arguments (for example, `__eq__`). ghstack-source-id: 312288551 @exported-using-ghexport Differential Revision: [D82873169](https://our.internmc.facebook.com/intern/diff/D82873169/)

facebook-github-bot · 2025-09-26T04:16:06Z

@swolchok has exported this pull request. If you are a Meta employee, you can view the originating diff in D82873169.

facebook-github-bot · 2025-09-26T17:51:54Z

@pytorchbot merge

(Initiating merge automatically since Phabricator Diff has merged)

pytorchmergebot · 2025-09-26T17:53:41Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

See comment on the macro definition. In short, pybind11 3.x added `py::native_enum`, and also had to add overhead for that new way to bind enums on the critical path for calling functions that take regular old `py::enum_`s as arguments (for example, `__eq__`). Differential Revision: [D82873169](https://our.internmc.facebook.com/intern/diff/D82873169/) Pull Request resolved: #163527 Approved by: https://github.com/ezyang

swolchok · 2025-10-02T15:39:47Z

@pytorchbot revert

pytorch-bot · 2025-10-02T15:39:50Z

❌ 🤖 pytorchbot command failed:

@pytorchbot revert: error: the following arguments are required: -m/--message, -c/--classification

usage: @pytorchbot revert -m MESSAGE -c
                          {nosignal,ignoredsignal,landrace,weird,ghfirst,autorevert}

Try @pytorchbot --help for more info.

swolchok · 2025-10-02T15:40:50Z

@pytorchbot revert -m "breaking import torch in debug builds, see #164297" -c nosignal

pytorchmergebot · 2025-10-02T15:42:27Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2025-10-02T15:42:46Z

@swolchok your PR has been successfully reverted.
⚠️ This PR might contain internal changes
cc: @pytorch/pytorch-dev-infra

This reverts commit 50c0550. Reverted #163527 on behalf of https://github.com/swolchok due to breaking import torch in debug builds, see #164297 ([comment](#163527 (comment)))

swolchok requested review from eqy, syed-ahmed, mikaylagawarecki, sraikund16, albanD and soulitzer as code owners September 22, 2025 16:41

pytorch-bot bot added ciflow/inductor module: dynamo oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category labels Sep 22, 2025

facebook-github-bot added fb-exported meta-exported oncall: jit Add this issue/PR to JIT oncall triage queue labels Sep 22, 2025

ezyang approved these changes Sep 24, 2025

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Sep 24, 2025

pytorchmergebot added the merging label Sep 26, 2025

pytorchmergebot closed this in 50c0550 Sep 26, 2025

pytorchmergebot added Merged and removed merging labels Sep 26, 2025

rtimpe mentioned this pull request Sep 30, 2025

Segfault trying to import torch #164297

Closed

pytorchmergebot added Reverted ci-no-td Do not run TD on this PR labels Oct 2, 2025

pytorchmergebot reopened this Oct 2, 2025

Add magic TORCH_MAKE_PYBIND_ENUM_FASTER macro #163527

Are you sure you want to change the base?

Add magic TORCH_MAKE_PYBIND_ENUM_FASTER macro #163527

Uh oh!

Conversation

swolchok commented Sep 22, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/163527

✅ No Failures

Uh oh!

facebook-github-bot commented Sep 22, 2025

Uh oh!

facebook-github-bot commented Sep 22, 2025

Uh oh!

Skylion007 commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

swolchok commented Sep 22, 2025

Uh oh!

Skylion007 commented Sep 22, 2025

Uh oh!

facebook-github-bot commented Sep 22, 2025

Uh oh!

facebook-github-bot commented Sep 23, 2025

Uh oh!

ezyang left a comment

Choose a reason for hiding this comment

Uh oh!

swolchok commented Sep 25, 2025

Uh oh!

swolchok commented Sep 25, 2025

Uh oh!

swolchok commented Sep 25, 2025

Uh oh!

facebook-github-bot commented Sep 26, 2025

Uh oh!

facebook-github-bot commented Sep 26, 2025

Uh oh!

pytorchmergebot commented Sep 26, 2025

Merge started

Uh oh!

swolchok commented Oct 2, 2025

Uh oh!

pytorch-bot bot commented Oct 2, 2025

Uh oh!

swolchok commented Oct 2, 2025

Uh oh!

pytorchmergebot commented Oct 2, 2025

Uh oh!

pytorchmergebot commented Oct 2, 2025

Uh oh!

Uh oh!

swolchok commented Sep 22, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Sep 22, 2025 •

edited

Loading

Skylion007 commented Sep 22, 2025 •

edited

Loading