[BE] Save attributes for CppCompileError for pickleing #160294

henrylhtsang · 2025-08-11T03:28:10Z

Stack from ghstack (oldest at bottom):

-> [BE] Save attributes for CppCompileError for pickleing #160294

Differential Revision: D79977408

Context:
When testing cutlass backend and used autotune with subproc, sometimes I would see C++ compilation error (expected) followed by

Traceback (most recent call last):
  File "/torch/_inductor/autotune_process.py", line 175, in get
    result = TuningProcess.recv(self.read_pipe)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/torch/_inductor/autotune_process.py", line 99, in recv
    return pickle.load(read_pipe)
           ^^^^^^^^^^^^^^^^^^^^^^
TypeError: CppCompileError.__init__() missing 1 required positional argument: 'output'

which is unexpected. After asking claude, it seems

Now I can see the issue. The CppCompileError class requires two arguments: cmd (a list of strings) and output (a string). However, when exceptions are being pickled and unpickled across process boundaries, the pickling process might not be preserving the constructor arguments correctly.

The problem is likely that when a CppCompileError is raised in the subprocess and then pickled/unpickled through the recv function, the unpickling process is trying to reconstruct the exception but doesn't have the required constructor arguments.

The issue is clear now. The CppCompileError class doesn't have custom pickle methods (__reduce__, __getstate__, __setstate__), so when it's pickled and unpickled across process boundaries, Python's default pickling mechanism tries to reconstruct it but fails because it doesn't preserve the constructor arguments properly.

The solution is to add a __reduce__ method to the CppCompileError class to ensure it can be properly pickled and unpickled. Let me implement this fix:

Adding these seem to help.

fbcode repro: D79977541

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben

Differential Revision: [D79977408](https://our.internmc.facebook.com/intern/diff/D79977408/) [ghstack-poisoned]

pytorch-bot · 2025-08-11T03:28:14Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/160294

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit e954654 with merge base eed9dbf ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Differential Revision: [D79977408](https://our.internmc.facebook.com/intern/diff/D79977408/) ghstack-source-id: 302028921 Pull Request resolved: #160294

facebook-github-bot · 2025-08-11T03:28:24Z

This pull request was exported from Phabricator. Differential Revision: D79977408

masnesral

LG; fix lint?

Differential Revision: [D79977408](https://our.internmc.facebook.com/intern/diff/D79977408/) Context: When testing cutlass backend and used autotune with subproc, sometimes I would see C++ compilation error (expected) followed by ``` Traceback (most recent call last): File "/torch/_inductor/autotune_process.py", line 175, in get result = TuningProcess.recv(self.read_pipe) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/torch/_inductor/autotune_process.py", line 99, in recv return pickle.load(read_pipe) ^^^^^^^^^^^^^^^^^^^^^^ TypeError: CppCompileError.__init__() missing 1 required positional argument: 'output' ``` which is unexpected. After asking claude, it seems > Now I can see the issue. The `CppCompileError` class requires two arguments: `cmd` (a list of strings) and `output` (a string). However, when exceptions are being pickled and unpickled across process boundaries, the pickling process might not be preserving the constructor arguments correctly. > > The problem is likely that when a `CppCompileError` is raised in the subprocess and then pickled/unpickled through the `recv` function, the unpickling process is trying to reconstruct the exception but doesn't have the required constructor arguments. > > The issue is clear now. The `CppCompileError` class doesn't have custom pickle methods (`__reduce__`, `__getstate__`, `__setstate__`), so when it's pickled and unpickled across process boundaries, Python's default pickling mechanism tries to reconstruct it but fails because it doesn't preserve the constructor arguments properly. > > The solution is to add a `__reduce__` method to the `CppCompileError` class to ensure it can be properly pickled and unpickled. Let me implement this fix: Adding these seem to help. fbcode repro: [D79977541](https://www.internalfb.com/diff/D79977541) cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov coconutruben [ghstack-poisoned]

Pull Request resolved: #160294 ghstack-source-id: 302162331 @exported-using-ghexport Differential Revision: [D79977408](https://our.internmc.facebook.com/intern/diff/D79977408/)

facebook-github-bot · 2025-08-11T17:48:49Z

This pull request was exported from Phabricator. Differential Revision: D79977408

henrylhtsang · 2025-08-12T01:39:21Z

@pytorchbot merge -r

pytorchmergebot · 2025-08-12T01:40:53Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

[ghstack-poisoned]

pytorchmergebot · 2025-08-12T01:41:07Z

Successfully rebased gh/henrylhtsang/142/orig onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/160294)

Pull Request resolved: #160294 ghstack-source-id: 08c3f3c @exported-using-ghexport Differential Revision: [D79977408](https://our.internmc.facebook.com/intern/diff/D79977408/)

pytorchmergebot · 2025-08-12T01:42:26Z

Merge failed

Reason: This PR has internal changes and must be landed via Phabricator! Please try reimporting/rexporting the PR!

Details for Dev Infra team

Raised by workflow job

Differential Revision: [D79977408](https://our.internmc.facebook.com/intern/diff/D79977408/) Context: When testing cutlass backend and used autotune with subproc, sometimes I would see C++ compilation error (expected) followed by ``` Traceback (most recent call last): File "/torch/_inductor/autotune_process.py", line 175, in get result = TuningProcess.recv(self.read_pipe) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/torch/_inductor/autotune_process.py", line 99, in recv return pickle.load(read_pipe) ^^^^^^^^^^^^^^^^^^^^^^ TypeError: CppCompileError.__init__() missing 1 required positional argument: 'output' ``` which is unexpected. After asking claude, it seems > Now I can see the issue. The `CppCompileError` class requires two arguments: `cmd` (a list of strings) and `output` (a string). However, when exceptions are being pickled and unpickled across process boundaries, the pickling process might not be preserving the constructor arguments correctly. > > The problem is likely that when a `CppCompileError` is raised in the subprocess and then pickled/unpickled through the `recv` function, the unpickling process is trying to reconstruct the exception but doesn't have the required constructor arguments. > > The issue is clear now. The `CppCompileError` class doesn't have custom pickle methods (`__reduce__`, `__getstate__`, `__setstate__`), so when it's pickled and unpickled across process boundaries, Python's default pickling mechanism tries to reconstruct it but fails because it doesn't preserve the constructor arguments properly. > > The solution is to add a `__reduce__` method to the `CppCompileError` class to ensure it can be properly pickled and unpickled. Let me implement this fix: Adding these seem to help. fbcode repro: [D79977541](https://www.internalfb.com/diff/D79977541) cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov coconutruben [ghstack-poisoned]

Pull Request resolved: #160294 ghstack-source-id: 302451054 @exported-using-ghexport Differential Revision: [D79977408](https://our.internmc.facebook.com/intern/diff/D79977408/)

facebook-github-bot · 2025-08-12T17:20:52Z

This pull request was exported from Phabricator. Differential Revision: D79977408

henrylhtsang · 2025-08-12T21:52:52Z

@pytorchbot merge

pytorchmergebot · 2025-08-12T21:54:59Z

Merge failed

Reason: This PR needs a release notes: label
If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Details for Dev Infra team

Raised by workflow job

henrylhtsang · 2025-08-12T21:59:30Z

@pytorchbot merge

pytorchmergebot · 2025-08-12T22:03:13Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Differential Revision: [D79977408](https://our.internmc.facebook.com/intern/diff/D79977408/) Context: When testing cutlass backend and used autotune with subproc, sometimes I would see C++ compilation error (expected) followed by ``` Traceback (most recent call last): File "/torch/_inductor/autotune_process.py", line 175, in get result = TuningProcess.recv(self.read_pipe) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/torch/_inductor/autotune_process.py", line 99, in recv return pickle.load(read_pipe) ^^^^^^^^^^^^^^^^^^^^^^ TypeError: CppCompileError.__init__() missing 1 required positional argument: 'output' ``` which is unexpected. After asking claude, it seems > Now I can see the issue. The `CppCompileError` class requires two arguments: `cmd` (a list of strings) and `output` (a string). However, when exceptions are being pickled and unpickled across process boundaries, the pickling process might not be preserving the constructor arguments correctly. > > The problem is likely that when a `CppCompileError` is raised in the subprocess and then pickled/unpickled through the `recv` function, the unpickling process is trying to reconstruct the exception but doesn't have the required constructor arguments. > > The issue is clear now. The `CppCompileError` class doesn't have custom pickle methods (`__reduce__`, `__getstate__`, `__setstate__`), so when it's pickled and unpickled across process boundaries, Python's default pickling mechanism tries to reconstruct it but fails because it doesn't preserve the constructor arguments properly. > > The solution is to add a `__reduce__` method to the `CppCompileError` class to ensure it can be properly pickled and unpickled. Let me implement this fix: Adding these seem to help. fbcode repro: [D79977541](https://www.internalfb.com/diff/D79977541) Pull Request resolved: #160294 Approved by: https://github.com/masnesral

Differential Revision: [D79977408](https://our.internmc.facebook.com/intern/diff/D79977408/) Context: When testing cutlass backend and used autotune with subproc, sometimes I would see C++ compilation error (expected) followed by ``` Traceback (most recent call last): File "/torch/_inductor/autotune_process.py", line 175, in get result = TuningProcess.recv(self.read_pipe) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/torch/_inductor/autotune_process.py", line 99, in recv return pickle.load(read_pipe) ^^^^^^^^^^^^^^^^^^^^^^ TypeError: CppCompileError.__init__() missing 1 required positional argument: 'output' ``` which is unexpected. After asking claude, it seems > Now I can see the issue. The `CppCompileError` class requires two arguments: `cmd` (a list of strings) and `output` (a string). However, when exceptions are being pickled and unpickled across process boundaries, the pickling process might not be preserving the constructor arguments correctly. > > The problem is likely that when a `CppCompileError` is raised in the subprocess and then pickled/unpickled through the `recv` function, the unpickling process is trying to reconstruct the exception but doesn't have the required constructor arguments. > > The issue is clear now. The `CppCompileError` class doesn't have custom pickle methods (`__reduce__`, `__getstate__`, `__setstate__`), so when it's pickled and unpickled across process boundaries, Python's default pickling mechanism tries to reconstruct it but fails because it doesn't preserve the constructor arguments properly. > > The solution is to add a `__reduce__` method to the `CppCompileError` class to ensure it can be properly pickled and unpickled. Let me implement this fix: Adding these seem to help. fbcode repro: [D79977541](https://www.internalfb.com/diff/D79977541) Pull Request resolved: pytorch#160294 Approved by: https://github.com/masnesral

[BE] Save attributes for pickleing

b264446

Differential Revision: [D79977408](https://our.internmc.facebook.com/intern/diff/D79977408/) [ghstack-poisoned]

henrylhtsang added a commit that referenced this pull request Aug 11, 2025

[BE] Save attributes for pickleing

f17afd6

Differential Revision: [D79977408](https://our.internmc.facebook.com/intern/diff/D79977408/) ghstack-source-id: 302028921 Pull Request resolved: #160294

pytorch-bot bot added ciflow/inductor module: inductor labels Aug 11, 2025

facebook-github-bot added the fb-exported label Aug 11, 2025

henrylhtsang requested a review from masnesral August 11, 2025 03:28

henrylhtsang changed the title ~~[BE] Save attributes for pickleing~~ [BE] Save attributes for CppCompileError for pickleing Aug 11, 2025

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Aug 11, 2025

masnesral approved these changes Aug 11, 2025

View reviewed changes

Update

93a4526

[ghstack-poisoned]

pytorchmergebot added the merging label Aug 12, 2025

pytorchmergebot removed the merging label Aug 12, 2025

pytorchmergebot added the merging label Aug 12, 2025

pytorchmergebot removed the merging label Aug 12, 2025

henrylhtsang added the topic: not user facing topic category label Aug 12, 2025

pytorchmergebot added the merging label Aug 12, 2025

pytorchmergebot added the Merged label Aug 12, 2025

pytorchmergebot closed this in 101276f Aug 12, 2025

pytorchmergebot removed the merging label Aug 12, 2025

github-actions bot deleted the gh/henrylhtsang/142/head branch September 12, 2025 02:08

Uh oh!

[BE] Save attributes for CppCompileError for pickleing #160294

[BE] Save attributes for CppCompileError for pickleing #160294

Uh oh!

Conversation

henrylhtsang commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/160294

✅ No Failures

Uh oh!

facebook-github-bot commented Aug 11, 2025

Uh oh!

masnesral left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Aug 11, 2025

Uh oh!

henrylhtsang commented Aug 12, 2025

Uh oh!

pytorchmergebot commented Aug 12, 2025

Uh oh!

pytorchmergebot commented Aug 12, 2025

Uh oh!

pytorchmergebot commented Aug 12, 2025

Merge failed

Uh oh!

facebook-github-bot commented Aug 12, 2025

Uh oh!

henrylhtsang commented Aug 12, 2025

Uh oh!

pytorchmergebot commented Aug 12, 2025

Merge failed

Uh oh!

henrylhtsang commented Aug 12, 2025

Uh oh!

pytorchmergebot commented Aug 12, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

henrylhtsang commented Aug 11, 2025 •

edited

Loading

pytorch-bot bot commented Aug 11, 2025 •

edited

Loading