fix inductor logging for torch._scaled_mm #149769

vkuzo · 2025-03-21T22:32:46Z

Stack from ghstack (oldest at bottom):

-> fix inductor logging for torch._scaled_mm #149769

Summary:

#148800 made inductor logs
throw an exception of the model contains torch._scaled_mm because
it split a string by underscore and making some assumptions about the resulting
list, and torch._scaled_mm has underscores which breaks those
assumptions.

Fixing by making the string parsing code slightly less brittle and
adding a test.

Test Plan:

pytest test/dynamo/test_logging.py -s -k test_inductor_mm_info
pytest test/dynamo/test_logging.py -s -k test_inductor_scaled_mm_info

Reviewers:

Subscribers:

Tasks:

Tags:

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov

[ghstack-poisoned]

pytorch-bot · 2025-03-21T22:32:49Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/149769

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (4 Unrelated Failures)

As of commit bfa3c1d with merge base 23183fe ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 1, 2, linux.8xlarge.amx) (gh) (similar failure)
detectron2_fcos_r_50_fpn

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / linux-jammy-xpu-2025.0-py3.9 / build (gh) (trunk failure)
/usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/sstream:152:52: error: expected value in expression

UNSTABLE - The following jobs are marked as unstable, possibly due to flakiness on trunk:

pull / cuda12.4-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu) (gh) (#149370)
Process completed with exit code 1.
pull / linux-jammy-py3-clang12-executorch / test (executorch, 1, 1, linux.2xlarge) (gh) (#144480)
backends/xnnpack/test/passes/test_convert_to_linear.py::TestConvertToLinear::test_fp32_convert_to_linear

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ghstack-poisoned]

Summary: #148800 made inductor logs throw an exception of the model contains `torch._scaled_mm` because it split a string by underscore and making some assumptions about the resulting list, and `torch._scaled_mm` has underscores which breaks those assumptions. Fixing by making the string parsing code slightly less brittle and adding a test. Test Plan: ```bash pytest test/dynamo/test_logging.py -s -k test_inductor_mm_info pytest test/dynamo/test_logging.py -s -k test_inductor_scaled_mm_info ``` Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 7d9e184 Pull Request resolved: #149769

YUNQIUGUO

thanks!!

vkuzo · 2025-03-21T23:25:51Z

@pytorchbot merge

pytorchmergebot · 2025-03-21T23:27:37Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-03-21T23:59:35Z

Merge failed

Reason: 1 jobs have failed, first few of them are: inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_torchbench, 1, 2, linux.8xlarge.amx)

Details for Dev Infra team

Raised by workflow job

Skylion007 · 2025-03-23T19:30:57Z

torch/_inductor/compile_fx.py

-    # This message is for printing overview information of inductor mm counts, shapes,etc after lowering
-    if log.isEnabledFor(logging.INFO):
+    # This message is for printing overview information of inductor mm counts, shapes, etc after lowering
+    if log.isEnabledFor(logging.INFO) and len(counters["aten_mm_info"].items()):


No need for item call? Check if the keys are empty is sufficient?

Skylion007 · 2025-03-23T19:31:49Z

torch/_inductor/compile_fx.py

+            # below is a bit brittle, but should work
+            parts = key.split("_")
+            if len(parts) < 4:
+                log.info(f"Unable to parse key {key}")  # noqa: G004


Just use %s logging instead of fstring to properly clear the error. Since it's already in an f statement though it's fine.

Skylion007 · 2025-03-23T19:31:49Z

torch/_inductor/compile_fx.py

+            # below is a bit brittle, but should work
+            parts = key.split("_")
+            if len(parts) < 4:
+                log.info(f"Unable to parse key {key}")  # noqa: G004


Just use %s logging instead of fstring to properly clear the error. Since it's already in an f statement though it's fine.

[ghstack-poisoned]

Summary: #148800 made inductor logs throw an exception of the model contains `torch._scaled_mm` because it split a string by underscore and making some assumptions about the resulting list, and `torch._scaled_mm` has underscores which breaks those assumptions. Fixing by making the string parsing code slightly less brittle and adding a test. Test Plan: ```bash pytest test/dynamo/test_logging.py -s -k test_inductor_mm_info pytest test/dynamo/test_logging.py -s -k test_inductor_scaled_mm_info ``` Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: a6e2865 Pull Request resolved: #149769

[ghstack-poisoned]

Summary: #148800 made inductor logs throw an exception of the model contains `torch._scaled_mm` because it split a string by underscore and making some assumptions about the resulting list, and `torch._scaled_mm` has underscores which breaks those assumptions. Fixing by making the string parsing code slightly less brittle and adding a test. Test Plan: ```bash pytest test/dynamo/test_logging.py -s -k test_inductor_mm_info pytest test/dynamo/test_logging.py -s -k test_inductor_scaled_mm_info ``` Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 425ede2 Pull Request resolved: #149769

Summary: This pr is just for recreation of the original pr: pytorch#149769 Fix for `torch._scaled_mm` op, which breaks the original parsing assumptions. Test Plan: CI Differential Revision: D71828732

Summary: This pr is just for recreation of the original pr: #149769 Fix for `torch._scaled_mm` op mm logging, which breaks the original brittle underscore parsing assumptions. Test Plan: CI Differential Revision: D71828732 Pull Request resolved: #149967 Approved by: https://github.com/vkuzo

eellison · 2025-04-15T19:20:43Z

torch/_inductor/compile_fx.py

-    # This message is for printing overview information of inductor mm counts, shapes,etc after lowering
-    if log.isEnabledFor(logging.INFO):
+    # This message is for printing overview information of inductor mm counts, shapes, etc after lowering
+    if log.isEnabledFor(logging.INFO) and len(counters["aten_mm_info"].items()):


Summary: This pr is just for recreation of the original pr: pytorch#149769 Fix for `torch._scaled_mm` op mm logging, which breaks the original brittle underscore parsing assumptions. Test Plan: CI Differential Revision: D71828732 Pull Request resolved: pytorch#149967 Approved by: https://github.com/vkuzo

github-actions · 2025-06-14T19:34:09Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

Update

de3a022

[ghstack-poisoned]

pytorch-bot bot added ciflow/inductor module: dynamo module: inductor labels Mar 21, 2025

Update

481695d

[ghstack-poisoned]

vkuzo requested review from YUNQIUGUO, drisspg and eellison March 21, 2025 22:36

vkuzo added topic: bug fixes topic category release notes: inductor labels Mar 21, 2025

YUNQIUGUO approved these changes Mar 21, 2025

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Mar 21, 2025

pytorchmergebot added the merging label Mar 21, 2025

pytorchmergebot removed the merging label Mar 21, 2025

Skylion007 reviewed Mar 23, 2025

View reviewed changes

Update

8febbd3

[ghstack-poisoned]

Update

bfa3c1d

[ghstack-poisoned]

YUNQIUGUO mentioned this pull request Mar 25, 2025

[inductor] Fix mm logging for torch._scaled_.mm #149967

Closed

eellison approved these changes Apr 15, 2025

View reviewed changes

github-actions bot added the Stale label Jun 14, 2025

github-actions bot closed this Jul 14, 2025

github-actions bot deleted the gh/vkuzo/10/head branch August 14, 2025 02:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix inductor logging for torch._scaled_mm #149769

fix inductor logging for torch._scaled_mm #149769

Uh oh!

vkuzo commented Mar 21, 2025 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Mar 21, 2025 •

edited

Loading

Uh oh!

YUNQIUGUO left a comment

Uh oh!

vkuzo commented Mar 21, 2025

Uh oh!

pytorchmergebot commented Mar 21, 2025

Uh oh!

pytorchmergebot commented Mar 21, 2025

Uh oh!

Skylion007 Mar 23, 2025

Uh oh!

eellison Apr 15, 2025

Uh oh!

Skylion007 Mar 23, 2025

Uh oh!

Skylion007 Mar 23, 2025

Uh oh!

eellison Apr 15, 2025

Uh oh!

github-actions bot commented Jun 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

fix inductor logging for torch._scaled_mm #149769

fix inductor logging for torch._scaled_mm #149769

Uh oh!

Conversation

vkuzo commented Mar 21, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/149769

✅ You can merge normally! (4 Unrelated Failures)

Uh oh!

YUNQIUGUO left a comment

Choose a reason for hiding this comment

Uh oh!

vkuzo commented Mar 21, 2025

Uh oh!

pytorchmergebot commented Mar 21, 2025

Merge started

Uh oh!

pytorchmergebot commented Mar 21, 2025

Merge failed

Uh oh!

Skylion007 Mar 23, 2025

Choose a reason for hiding this comment

Uh oh!

eellison Apr 15, 2025

Choose a reason for hiding this comment

Uh oh!

Skylion007 Mar 23, 2025

Choose a reason for hiding this comment

Uh oh!

Skylion007 Mar 23, 2025

Choose a reason for hiding this comment

Uh oh!

eellison Apr 15, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

vkuzo commented Mar 21, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Mar 21, 2025 •

edited

Loading