[symbolic shapes] if symbol not in var_ranges default to unknown range #127681

ColinPeppler · 2024-06-01T05:31:37Z

Purpose of this PR is to get around this error: #127677

Differential Revision: D58048558

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @amjames @desertfire @chauhang

pytorch-bot · 2024-06-01T05:31:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/127681

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit a4564f4 with merge base 2e77916 ():

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#126884)
cspdarknet53
inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (inductor_torchbench_cpu_smoketest_perf, 1, 1, linux.24xl.spr-metal, unstable) (gh) (#126993)
Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2024-06-01T05:31:49Z

This pull request was exported from Phabricator. Differential Revision: D58048558

Skylion007 · 2024-06-01T13:59:21Z

torch/_inductor/index_propagation.py

+                    )
+                return expr
+            except KeyError as e:
+                log.warn(str(e))


Use 'log.warning("", exc_info=True)'

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

facebook-github-bot · 2024-06-01T16:35:54Z

This pull request was exported from Phabricator. Differential Revision: D58048558

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

facebook-github-bot · 2024-06-01T16:36:58Z

This pull request was exported from Phabricator. Differential Revision: D58048558

lezcano

I would really like to understand #127677 (comment), as this is just a hint that there's another bug lurking underneath.

ColinPeppler · 2024-06-01T16:42:57Z

@lezcano
I'd love to take a look, but I also need to urgently resolve an error happening internally. Can we slap a bandaid now and I'll return to this ASAP?

lezcano

Sure, let's land this as a hotfix and leave the other issue open.

lezcano · 2024-06-01T16:46:14Z

torch/fx/experimental/symbolic_shapes.py

+                vr = self.var_to_range[k]
+            except KeyError:
+                log.warning(f"{k} is not in var_to_range, defaulting to unknown range.")
+                vr = ValueRanges(-sys.maxsize, sys.maxsize)


Use self._default_unspecified_value_range() then, as this one has an off-by-one error.

ColinPeppler · 2024-06-01T17:02:29Z

sounds good, assigned myself to the issue

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

facebook-github-bot · 2024-06-01T17:06:51Z

This pull request was exported from Phabricator. Differential Revision: D58048558

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

facebook-github-bot · 2024-06-01T17:08:18Z

This pull request was exported from Phabricator. Differential Revision: D58048558

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

facebook-github-bot · 2024-06-01T21:49:01Z

This pull request was exported from Phabricator. Differential Revision: D58048558

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

facebook-github-bot · 2024-06-01T21:50:01Z

This pull request was exported from Phabricator. Differential Revision: D58048558

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

facebook-github-bot · 2024-06-01T22:01:17Z

This pull request was exported from Phabricator. Differential Revision: D58048558

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

facebook-github-bot · 2024-06-01T22:02:19Z

This pull request was exported from Phabricator. Differential Revision: D58048558

facebook-github-bot · 2024-06-02T02:26:28Z

@pytorchbot merge -f 'Landed internally'

(Initiating merge automatically since Phabricator Diff has merged, using force because this PR might not pass merge_rules.json but landed internally)

pytorchmergebot · 2024-06-02T02:28:15Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

bhack · 2024-06-05T11:52:11Z

With recently nighlty I've just got many of this warning.

W0605 11:48:51.002000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] q0 is not in var_ranges, defaulting to unknown range.
W0605 11:48:51.002000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] q1 is not in var_ranges, defaulting to unknown range.
W0605 11:48:51.013000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] q0 is not in var_ranges, defaulting to unknown range.
W0605 11:48:51.013000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] q1 is not in var_ranges, defaulting to unknown range.
W0605 11:48:51.324000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] z1 is not in var_ranges, defaulting to unknown range.
W0605 11:48:51.324000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] z0 is not in var_ranges, defaulting to unknown range.
W0605 11:48:51.334000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] z1 is not in var_ranges, defaulting to unknown range.
W0605 11:48:51.334000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] z0 is not in var_ranges, defaulting to unknown range.
W0605 11:48:52.669000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] q0 is not in var_ranges, defaulting to unknown range.
W0605 11:48:52.800000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] z0 is not in var_ranges, defaulting to unknown range.
W0605 11:48:54.093000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] q0 is not in var_ranges, defaulting to unknown range.
W0605 11:48:54.093000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] q1 is not in var_ranges, defaulting to unknown range.
W0605 11:48:54.152000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] z1 is not in var_ranges, defaulting to unknown range.
W0605 11:48:54.152000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] z0 is not in var_ranges, defaulting to unknown range.
W0605 11:48:54.734000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] q0 is not in var_ranges, defaulting to unknown range.
W0605 11:48:54.869000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] z0 is not in var_ranges, defaulting to unknown range.
W0605 11:48:56.124000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] q0 is not in var_ranges, defaulting to unknown range.
W0605 11:48:56.124000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] q1 is not in var_ranges, defaulting to unknown range.
W0605 11:48:56.189000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] z1 is not in var_ranges, defaulting to unknown range.
W0605 11:48:56.189000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] z0 is not in var_ranges, defaulting to unknown range.
W0605 11:48:57.112000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] q0 is not in var_ranges, defaulting to unknown range.
W0605 11:48:57.240000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] z0 is not in var_ranges, defaulting to unknown range.
W0605 11:49:03.713000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] y1 is not in var_ranges, defaulting to unknown range.
W0605 11:49:03.713000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] y0 is not in var_ranges, defaulting to unknown range.
W0605 11:49:03.725000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] y1 is not in var_ranges, defaulting to unknown range.
W0605 11:49:03.725000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] y0 is not in var_ranges, defaulting to unknown range.
W0605 11:49:04.647000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] x0 is not in var_ranges, defaulting to unknown range.
W0605 11:49:05.445000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] x2 is not in var_ranges, defaulting to unknown range.
W0605 11:49:05.445000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] x1 is not in var_ranges, defaulting to unknown range.
W0605 11:49:05.923000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] x0 is not in var_ranges, defaulting to unknown range.
W0605 11:49:06.703000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] x2 is not in var_ranges, defaulting to unknown range.
W0605 11:49:06.703000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] x1 is not in var_ranges, defaulting to unknown range.
W0605 11:49:07.186000 134494347568960 torch/fx/experimental/symbolic_shapes.py:4417] [0/0] x0 is not in var_ranges, defaulting to unknown range.

Is there any user action item on these?

lezcano · 2024-06-05T13:00:24Z

@ColinPeppler will look into how to properly fix #127677

pytorch#127681) Purpose of this PR is to get around this error: pytorch#127677 Differential Revision: D58048558 Pull Request resolved: pytorch#127681 Approved by: https://github.com/lezcano

ColinPeppler · 2024-06-06T00:41:19Z

@bhack those warnings would've led to failures. And if you have a minimal repro, pls share in #127677 then I can check with the fix.

bhack · 2024-06-06T00:44:39Z

@ColinPeppler The code is quite large to isolate a minimal standalone to run. I have the full env in another ticket if you want.
If the full env is too much for you do you need somethin from https://pytorch.org/docs/stable/torch.compiler_troubleshooting.html ?

lezcano · 2024-06-06T07:47:06Z

Could you try to isolate a smaller repro by putting breakpoints on this warnign and seeing the code that calls it @bhack and doing some manual binary search in the resulting fx graph?

bhack · 2024-06-06T13:37:35Z

@lezcano A first isolation it is at #127677 (comment)

jgong5 · 2024-06-08T08:21:20Z

torch/fx/experimental/symbolic_shapes.py

+            try:
+                vr = var_ranges[k]
+            except KeyError:
+                log.warning("%s is not in var_ranges, defaulting to unknown range.", k)


May I know why it is a warning here? Would it always warn on unbacked symbols? The cpp template uses some unbacked symbols to build IR nodes for codegen which would always get warning here.

pytorch/torch/_inductor/codegen/cpp_gemm_template.py

Line 110 in 695502c

{%- set tile_X = kernel.slice_nd(X, [("m_start", "m_end"), ("k_start", "k_end")]) %}

lezcano · 2024-06-08T12:06:35Z

@ColinPeppler this is annoying too many people. Can you send a patch that removes the warning, or send a fix altogether?

ColinPeppler · 2024-06-10T17:01:27Z

sorry if I can't post a fix by end-of-today, I will remove the warning.

pytorch-bot bot added ciflow/inductor module: inductor labels Jun 1, 2024

facebook-github-bot added the fb-exported label Jun 1, 2024

ColinPeppler added the topic: not user facing topic category label Jun 1, 2024

Skylion007 reviewed Jun 1, 2024

View reviewed changes

ColinPeppler changed the title ~~[inductor] skip index propagation if there's a keyerror~~ [symbolic shapes] if symbol not in var_to_range default to unknown range Jun 1, 2024

ColinPeppler force-pushed the export-D58048558 branch from ec27fea to d06fe3f Compare June 1, 2024 16:35

ColinPeppler added a commit to ColinPeppler/pytorch that referenced this pull request Jun 1, 2024

[symbolic shapes] if symbol not in var_to_range default to unknown ra…

d06fe3f

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

ColinPeppler requested review from ezyang, lezcano and houseroad June 1, 2024 16:36

ColinPeppler added a commit to ColinPeppler/pytorch that referenced this pull request Jun 1, 2024

[symbolic shapes] if symbol not in var_to_range default to unknown ra…

a628b72

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

ColinPeppler force-pushed the export-D58048558 branch from d06fe3f to a628b72 Compare June 1, 2024 16:36

lezcano requested changes Jun 1, 2024

View reviewed changes

lezcano reviewed Jun 1, 2024

View reviewed changes

ColinPeppler force-pushed the export-D58048558 branch from a628b72 to 106e7c5 Compare June 1, 2024 17:06

ColinPeppler added a commit to ColinPeppler/pytorch that referenced this pull request Jun 1, 2024

[symbolic shapes] if symbol not in var_to_range default to unknown ra…

106e7c5

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

ColinPeppler requested a review from lezcano June 1, 2024 17:06

lezcano approved these changes Jun 1, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 1, 2024

ColinPeppler added a commit to ColinPeppler/pytorch that referenced this pull request Jun 1, 2024

[symbolic shapes] if symbol not in var_to_range default to unknown ra…

0f7b058

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

ColinPeppler force-pushed the export-D58048558 branch from 106e7c5 to 0f7b058 Compare June 1, 2024 17:08

ColinPeppler added a commit to ColinPeppler/pytorch that referenced this pull request Jun 1, 2024

[symbolic shapes] if symbol not in var_to_range default to unknown ra…

0a23541

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

ColinPeppler force-pushed the export-D58048558 branch from 83b78c0 to 0a23541 Compare June 1, 2024 21:48

ColinPeppler added a commit to ColinPeppler/pytorch that referenced this pull request Jun 1, 2024

[symbolic shapes] if symbol not in var_to_range default to unknown ra…

934b94c

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

ColinPeppler force-pushed the export-D58048558 branch from 0a23541 to 934b94c Compare June 1, 2024 21:49

ColinPeppler added a commit to ColinPeppler/pytorch that referenced this pull request Jun 1, 2024

[symbolic shapes] if symbol not in var_to_range default to unknown ra…

7656231

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

ColinPeppler force-pushed the export-D58048558 branch from 934b94c to 7656231 Compare June 1, 2024 22:01

[symbolic shapes] if symbol not in var_to_range default to unknown ra…

a4564f4

…nge (pytorch#127681) Summary: Issue: pytorch#127677 Test Plan: ci --- Differential Revision: D58048558

ColinPeppler force-pushed the export-D58048558 branch from 7656231 to a4564f4 Compare June 1, 2024 22:02

ColinPeppler changed the title ~~[symbolic shapes] if symbol not in var_to_range default to unknown range~~ [symbolic shapes] if symbol not in var_ranges default to unknown range Jun 2, 2024

pytorchmergebot added the merging label Jun 2, 2024

pytorchmergebot closed this in 16578e8 Jun 2, 2024

pytorchmergebot added Merged and removed merging labels Jun 2, 2024

jgong5 reviewed Jun 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[symbolic shapes] if symbol not in var_ranges default to unknown range #127681

[symbolic shapes] if symbol not in var_ranges default to unknown range #127681

ColinPeppler commented Jun 1, 2024 •

edited

Loading

pytorch-bot bot commented Jun 1, 2024 •

edited

Loading

facebook-github-bot commented Jun 1, 2024

Skylion007 Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

lezcano left a comment

ColinPeppler commented Jun 1, 2024

lezcano left a comment

lezcano Jun 1, 2024

ColinPeppler commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 2, 2024

pytorchmergebot commented Jun 2, 2024

bhack commented Jun 5, 2024

lezcano commented Jun 5, 2024

ColinPeppler commented Jun 6, 2024

bhack commented Jun 6, 2024 •

edited

Loading

lezcano commented Jun 6, 2024

bhack commented Jun 6, 2024

jgong5 Jun 8, 2024

lezcano commented Jun 8, 2024

ColinPeppler commented Jun 10, 2024

[symbolic shapes] if symbol not in var_ranges default to unknown range #127681

[symbolic shapes] if symbol not in var_ranges default to unknown range #127681

Conversation

ColinPeppler commented Jun 1, 2024 • edited Loading

pytorch-bot bot commented Jun 1, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/127681

✅ You can merge normally! (2 Unrelated Failures)

facebook-github-bot commented Jun 1, 2024

Skylion007 Jun 1, 2024

Choose a reason for hiding this comment

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

lezcano left a comment

Choose a reason for hiding this comment

ColinPeppler commented Jun 1, 2024

lezcano left a comment

Choose a reason for hiding this comment

lezcano Jun 1, 2024

Choose a reason for hiding this comment

ColinPeppler commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 1, 2024

facebook-github-bot commented Jun 2, 2024

pytorchmergebot commented Jun 2, 2024

Merge started

bhack commented Jun 5, 2024

lezcano commented Jun 5, 2024

ColinPeppler commented Jun 6, 2024

bhack commented Jun 6, 2024 • edited Loading

lezcano commented Jun 6, 2024

bhack commented Jun 6, 2024

jgong5 Jun 8, 2024

Choose a reason for hiding this comment

lezcano commented Jun 8, 2024

ColinPeppler commented Jun 10, 2024

ColinPeppler commented Jun 1, 2024 •

edited

Loading

pytorch-bot bot commented Jun 1, 2024 •

edited

Loading

bhack commented Jun 6, 2024 •

edited

Loading