Nicer logs for dynamic shapes #99277

voznesenskym · 2023-04-16T20:57:47Z

Stack from ghstack (oldest at bottom):

-> Nicer logs for dynamic shapes #99277

cc @soumith @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire

[ghstack-poisoned]

pytorch-bot · 2023-04-16T20:57:49Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/99277

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

ROCm jobs are failing with No space left on device

✅ No Failures

As of commit 1ecf54e:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 1f54111b681f46459d96358c50aa7460b1ff8d45 Pull Request resolved: #99277

voznesenskym · 2023-04-16T20:58:27Z

torch/_dynamo/variables/builder.py

@@ -1176,18 +1180,34 @@ def wrap_to_fake_tensor_and_record(
            # If there is no entry for this source, add the tensor to frame state with its current static size.
            # E.g., {} -> {“x”: [2, 4]}
            curr_sizes = list(e.size())
+            log.debug("Registered static shapes %s for %s", curr_sizes, name)


This one gets very noisy, so maybe we keep him debug, and make the others info? Open to bikshedding. @ezyang opine please.

It is hard to say without actually using the logs. Keeping everything as debug to start is a safe start, assuming this doesn't trigger too much (but I don't think it does, since we debug trace every opcode in dynamo). Note that you can selectively up logging from ONLY this module with TORCH_LOGS=+torch._dynamo.variables.builder

I would bikeshed the message here though; "registered" isn't very descriptive. In the log message, I think I probably want the words frame_state in it. Maybe frame_state[name] = curr_sizes. But in that case, why don't we move the log message down to where we actually do the setter, and then delete all the other branches?

cc soumith penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx desertfire [ghstack-poisoned]

ghstack-source-id: 48665ec78aa53fcb7434b2f92b0625baf7fb5ab3 Pull Request resolved: #99277 Better little none check

ezyang · 2023-04-17T02:38:20Z

torch/fx/experimental/symbolic_shapes.py

@@ -1945,10 +1945,10 @@ def hint():
                    msg = f"  {len(error_msgs) + 1}. {msg()}"
                    error_msgs.append(msg)
            if len(error_msgs) > 0:
-                log.warning("Warning only constraints violated %s", warn_msgs)
+                log.info("Warning only constraints violated %s", warn_msgs)


Wait, this looks like a botched merge, we shouldn't warn here at all, we're going to raise an error!

ezyang · 2023-04-17T02:38:40Z

torch/fx/experimental/symbolic_shapes.py

                raise ConstraintViolationError(f"Constraints violated!\n{error_msgs}")
            elif len(warn_msgs) > 0:
-                log.warning("%s Warning only constraints violated", len(warn_msgs))
+                log.info("%s Warning only constraints violated", len(warn_msgs))


This probably suppresses too much but we can figure out a better strategy later (maybe only warn this once per frame)

voznesenskym · 2023-04-24T06:07:23Z

@pytorchbot rebase

pytorchmergebot · 2023-04-24T06:09:21Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2023-04-24T06:09:28Z

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch rebase refs/remotes/origin/viable/strict gh/voznesenskym/102/orig returned non-zero exit code 1

warning: skipped previously applied commit cf387a5b65c
warning: skipped previously applied commit fb358849425
hint: use --reapply-cherry-picks to include skipped commits
hint: Disable this message with "git config advice.skippedCherryPicks false"
Rebasing (1/1)
Auto-merging torch/_dynamo/variables/builder.py
CONFLICT (content): Merge conflict in torch/_dynamo/variables/builder.py
Auto-merging torch/fx/experimental/symbolic_shapes.py
CONFLICT (content): Merge conflict in torch/fx/experimental/symbolic_shapes.py
error: could not apply 62d51eea02a... Nicer logs for dynamic shapes
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 62d51eea02a... Nicer logs for dynamic shapes

Raised by https://github.com/pytorch/pytorch/actions/runs/4783046149

cc soumith penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx desertfire [ghstack-poisoned]

ghstack-source-id: e62c51a2ad28a568699180f57613c165bdea7820 Pull Request resolved: #99277 Better little none check

voznesenskym · 2023-04-24T07:03:19Z

@pytorchbot merge

pytorchmergebot · 2023-04-24T07:05:09Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Nicer logs for dynamic shapes

ecfe3e8

[ghstack-poisoned]

pytorch-bot bot added the release notes: fx release notes category label Apr 16, 2023

voznesenskym added a commit that referenced this pull request Apr 16, 2023

Nicer logs for dynamic shapes

c27081c

ghstack-source-id: 1f54111b681f46459d96358c50aa7460b1ff8d45 Pull Request resolved: #99277

github-actions bot requested review from albanD, antoniojkim, bdhirsh, ezyang, jbschlosser, miladm, SherlockNoMad and wconstab April 16, 2023 20:57

github-actions bot added ciflow/inductor module: dynamo labels Apr 16, 2023

voznesenskym commented Apr 16, 2023

View reviewed changes

Update on "Nicer logs for dynamic shapes"

6602fc8

cc soumith penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx desertfire [ghstack-poisoned]

voznesenskym added a commit that referenced this pull request Apr 16, 2023

Nicer logs for dynamic shapes

62d51ee

ghstack-source-id: 48665ec78aa53fcb7434b2f92b0625baf7fb5ab3 Pull Request resolved: #99277 Better little none check

ezyang approved these changes Apr 17, 2023

View reviewed changes

ezyang reviewed Apr 17, 2023

View reviewed changes

albanD removed their request for review April 17, 2023 17:42

Update on "Nicer logs for dynamic shapes"

1ecf54e

cc soumith penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx desertfire [ghstack-poisoned]

voznesenskym added a commit that referenced this pull request Apr 24, 2023

Nicer logs for dynamic shapes

f662047

ghstack-source-id: e62c51a2ad28a568699180f57613c165bdea7820 Pull Request resolved: #99277 Better little none check

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Apr 24, 2023

pytorchmergebot added the merging label Apr 24, 2023

pytorchmergebot added the Merged label Apr 24, 2023

pytorchmergebot closed this in 466877b Apr 24, 2023

facebook-github-bot deleted the gh/voznesenskym/102/head branch June 8, 2023 18:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nicer logs for dynamic shapes #99277

Nicer logs for dynamic shapes #99277

voznesenskym commented Apr 16, 2023 •

edited

pytorch-bot bot commented Apr 16, 2023 •

edited

voznesenskym Apr 16, 2023

ezyang Apr 17, 2023

ezyang Apr 17, 2023

ezyang Apr 17, 2023

ezyang Apr 17, 2023

voznesenskym commented Apr 24, 2023

pytorchmergebot commented Apr 24, 2023

pytorchmergebot commented Apr 24, 2023

voznesenskym commented Apr 24, 2023

pytorchmergebot commented Apr 24, 2023

Nicer logs for dynamic shapes #99277

Nicer logs for dynamic shapes #99277

Conversation

voznesenskym commented Apr 16, 2023 • edited

pytorch-bot bot commented Apr 16, 2023 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/99277

❗ 1 Active SEVs

✅ No Failures

voznesenskym Apr 16, 2023

Choose a reason for hiding this comment

ezyang Apr 17, 2023

Choose a reason for hiding this comment

ezyang Apr 17, 2023

Choose a reason for hiding this comment

ezyang Apr 17, 2023

Choose a reason for hiding this comment

ezyang Apr 17, 2023

Choose a reason for hiding this comment

voznesenskym commented Apr 24, 2023

pytorchmergebot commented Apr 24, 2023

pytorchmergebot commented Apr 24, 2023

voznesenskym commented Apr 24, 2023

pytorchmergebot commented Apr 24, 2023

Merge started

voznesenskym commented Apr 16, 2023 •

edited

pytorch-bot bot commented Apr 16, 2023 •

edited