[inductor] Fix error from custom CUDA allocators #163422

jansel · 2025-09-20T21:28:26Z

Stack from ghstack (oldest at bottom):

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben

[ghstack-poisoned]

pytorch-bot · 2025-09-20T21:28:30Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/163422

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job

As of commit ca28793 with merge base 51152ef ():

CANCELLED JOB - The following job was cancelled. Please retry:

trunk / linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.gfx942.4) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Fixes #163257 ghstack-source-id: dad00db Pull-Request: #163422

Skylion007 · 2025-09-20T21:44:46Z

torch/_inductor/runtime/triton_heuristics.py

-        budget = torch.cuda.max_memory_allocated() - torch.cuda.memory_allocated()
+        try:
+            budget = torch.cuda.max_memory_allocated() - torch.cuda.memory_allocated()
+        except RuntimeError:


Don't even log what the runtime error was? Dangerous...

Look at the issue for context. I don't think spamming the user with warnings in that use case would be helpful.

[ghstack-poisoned]

eellison

Cudagraphs won't be useable with custom allocator, but yea, this should at least work outside of that case.

[ghstack-poisoned]

jansel · 2025-09-23T04:24:31Z

@pytorchbot merge

pytorchmergebot · 2025-09-23T04:26:29Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-09-23T10:24:53Z

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

jansel · 2025-09-23T15:34:03Z

@pytorchbot merge -i

pytorchmergebot · 2025-09-23T15:36:23Z

Merge started

Your change will be merged while ignoring the following 1 checks: trunk / linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.gfx942.4)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Fixes #163420 Pull Request resolved: #163481 Approved by: https://github.com/eellison ghstack dependencies: #163386, #163398, #163387, #163414, #163415, #163419, #163434, #163393, #163412, #163422

Fixes #163449 Pull Request resolved: #163520 Approved by: https://github.com/eellison ghstack dependencies: #163386, #163398, #163387, #163414, #163415, #163419, #163434, #163393, #163412, #163422, #163481

Fixes #163457 Pull Request resolved: #163482 Approved by: https://github.com/eellison ghstack dependencies: #163386, #163398, #163387, #163414, #163415, #163419, #163434, #163393, #163412, #163422, #163481, #163520

This reverts commit a8cd437. See #163481 (comment) This PR might also cause issues with cudagraphs. Pull Request resolved: #163737 Approved by: https://github.com/ezyang ghstack dependencies: #163386, #163398, #163387, #163414, #163415, #163419, #163434, #163393, #163412, #163422, #163481, #163520, #163482

Fixes pytorch#163257 Pull Request resolved: pytorch#163422 Approved by: https://github.com/eellison ghstack dependencies: pytorch#163386, pytorch#163398, pytorch#163387, pytorch#163414, pytorch#163415, pytorch#163419, pytorch#163434, pytorch#163393, pytorch#163412

Fixes pytorch#163420 Pull Request resolved: pytorch#163481 Approved by: https://github.com/eellison ghstack dependencies: pytorch#163386, pytorch#163398, pytorch#163387, pytorch#163414, pytorch#163415, pytorch#163419, pytorch#163434, pytorch#163393, pytorch#163412, pytorch#163422

Fixes pytorch#163449 Pull Request resolved: pytorch#163520 Approved by: https://github.com/eellison ghstack dependencies: pytorch#163386, pytorch#163398, pytorch#163387, pytorch#163414, pytorch#163415, pytorch#163419, pytorch#163434, pytorch#163393, pytorch#163412, pytorch#163422, pytorch#163481

Fixes pytorch#163457 Pull Request resolved: pytorch#163482 Approved by: https://github.com/eellison ghstack dependencies: pytorch#163386, pytorch#163398, pytorch#163387, pytorch#163414, pytorch#163415, pytorch#163419, pytorch#163434, pytorch#163393, pytorch#163412, pytorch#163422, pytorch#163481, pytorch#163520

This reverts commit a8cd437. See pytorch#163481 (comment) This PR might also cause issues with cudagraphs. Pull Request resolved: pytorch#163737 Approved by: https://github.com/ezyang ghstack dependencies: pytorch#163386, pytorch#163398, pytorch#163387, pytorch#163414, pytorch#163415, pytorch#163419, pytorch#163434, pytorch#163393, pytorch#163412, pytorch#163422, pytorch#163481, pytorch#163520, pytorch#163482

Fixes #163420 Pull Request resolved: #163481 Approved by: https://github.com/eellison ghstack dependencies: #163386, #163398, #163387, #163414, #163415, #163419, #163434, #163393, #163412, #163422

Fixes #163449 Pull Request resolved: #163520 Approved by: https://github.com/eellison ghstack dependencies: #163386, #163398, #163387, #163414, #163415, #163419, #163434, #163393, #163412, #163422, #163481

Fixes #163457 Pull Request resolved: #163482 Approved by: https://github.com/eellison ghstack dependencies: #163386, #163398, #163387, #163414, #163415, #163419, #163434, #163393, #163412, #163422, #163481, #163520

This reverts commit a8cd437. See #163481 (comment) This PR might also cause issues with cudagraphs. Pull Request resolved: #163737 Approved by: https://github.com/ezyang ghstack dependencies: #163386, #163398, #163387, #163414, #163415, #163419, #163434, #163393, #163412, #163422, #163481, #163520, #163482

Update

eaa0ca3

[ghstack-poisoned]

pytorch-bot bot added ciflow/inductor module: inductor labels Sep 20, 2025

This was referenced Sep 20, 2025

Better decomp for torch.eye #163386

Closed

[inductor] Support out_dtype arg to matmul #163393

Closed

[inductor] Fix bug where viewed outputs get padded #163398

Closed

jansel added a commit that referenced this pull request Sep 20, 2025

[inductor] Fix error from custom CUDA allocators

0c62c9b

Fixes #163257 ghstack-source-id: dad00db Pull-Request: #163422

Skylion007 reviewed Sep 20, 2025

View reviewed changes

jansel mentioned this pull request Sep 21, 2025

[inductor] Freeze layouts in FlexAttention #163434

Closed

jansel added the topic: not user facing topic category label Sep 21, 2025

jansel added 2 commits September 20, 2025 22:28

Update

72eecac

[ghstack-poisoned]

Update

d4faba3

[ghstack-poisoned]

This was referenced Sep 22, 2025

[inductor] Fix issue with scalar arg handling #163481

Closed

[inductor] Fix divmod error in decomp #163482

Closed

[inductor] Fix bugs in emulate_precision_casts #163520

Closed

jansel added 2 commits September 22, 2025 08:10

Update

f539923

[ghstack-poisoned]

Update

1e3281a

[ghstack-poisoned]

jansel requested a review from eellison September 22, 2025 15:12

eellison approved these changes Sep 22, 2025

View reviewed changes

Update

ca28793

[ghstack-poisoned]

jansel mentioned this pull request Sep 22, 2025

Update conv1d meta kernel to match eager #163584

Closed

jansel added the ciflow/trunk Trigger trunk jobs on your pull request label Sep 22, 2025

pytorchmergebot added the merging label Sep 23, 2025

pytorchmergebot added the Merged label Sep 23, 2025

pytorchmergebot closed this in 49e7b2f Sep 23, 2025

pytorchmergebot removed the merging label Sep 23, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[inductor] Fix error from custom CUDA allocators #163422

[inductor] Fix error from custom CUDA allocators #163422

Uh oh!

jansel commented Sep 20, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Sep 20, 2025 •

edited

Loading

Uh oh!

Skylion007 Sep 20, 2025

Uh oh!

jansel Sep 21, 2025

Uh oh!

eellison left a comment

Uh oh!

jansel commented Sep 23, 2025

Uh oh!

pytorchmergebot commented Sep 23, 2025

Uh oh!

pytorchmergebot commented Sep 23, 2025

Uh oh!

jansel commented Sep 23, 2025

Uh oh!

pytorchmergebot commented Sep 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[inductor] Fix error from custom CUDA allocators #163422

[inductor] Fix error from custom CUDA allocators #163422

Uh oh!

Conversation

jansel commented Sep 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/163422

❌ 1 Cancelled Job

Uh oh!

Skylion007 Sep 20, 2025

Choose a reason for hiding this comment

Uh oh!

jansel Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

eellison left a comment

Choose a reason for hiding this comment

Uh oh!

jansel commented Sep 23, 2025

Uh oh!

pytorchmergebot commented Sep 23, 2025

Merge started

Uh oh!

pytorchmergebot commented Sep 23, 2025

Uh oh!

jansel commented Sep 23, 2025

Uh oh!

pytorchmergebot commented Sep 23, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jansel commented Sep 20, 2025 •

edited

Loading

pytorch-bot bot commented Sep 20, 2025 •

edited

Loading