SetVariable in dynamo #103205

voznesenskym · 2023-06-07T21:36:16Z

Stack from ghstack (oldest at bottom):

-> SetVariable in dynamo #103205

Set initial
Fixes #94738

cc @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @aakhundov

Set initial [ghstack-poisoned]

pytorch-bot · 2023-06-07T21:36:18Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/103205

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure

As of commit fc038f9:

NEW FAILURE - The following job has failed:

cuda11.8-py3.10-gcc7-sm86 / test (inductor_torchbench_dynamic, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

cuda11.8-py3.10-gcc7-sm86 / test (inductor_torchbench, 1, 1, linux.g5.4xlarge.nvidia.gpu, unstable) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Set initial ghstack-source-id: 79873ff77503d7488c3b2f3929560fb5d3d069aa Pull Request resolved: #103205

torch/_dynamo/variables/builder.py

torch/_dynamo/variables/lists.py

torch/_dynamo/symbolic_convert.py

voznesenskym · 2023-06-08T17:32:43Z

torch/_dynamo/variables/lists.py

+        if sys.version_info >= (3, 11):
+            codegen.foreach(self.items)
+            return [create_instruction("BUILD_SET", arg=len(self.items))]
+        else:
+            create_load_global("set", True)
+            codegen.foreach(self.items)
+            return [create_instruction("CALL_FUNCTION", arg=len(self.items))]
+


I don't actually need this to be like this, I can simplify this to just build set.

torch/_dynamo/variables/lists.py

torch/_dynamo/symbolic_convert.py

Set initial cc penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx ipiszy [ghstack-poisoned]

Set initial ghstack-source-id: 030e4e6d7c5497c9f96ad9d36c94cdfeaf616761 Pull Request resolved: #103205 wip WIP WIP

Set initial cc penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx ipiszy [ghstack-poisoned]

Set initial ghstack-source-id: b7950036819956f2f0670b9ea278cda89d43bdff Pull Request resolved: #103205 wip WIP WIP WIP

Set initial cc penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx ipiszy [ghstack-poisoned]

voznesenskym · 2023-07-15T02:52:51Z

torch/_dynamo/variables/builder.py

+        if not self.tx.output.export:
+            # Export has (supposedly) valid cases for fake tensors as inputs here.
+            # I am not convinced, atm, but out of scope for what this assert was added for (protecting value checks
+            # in real_value_tensor_positive_aliases in the common case)
+            assert not isinstance(value, torch._subclasses.fake_tensor.FakeTensor)
+
+        if value in self.tx.output.real_value_tensor_positive_aliases:
+            stored_value = self.tx.output.real_value_tensor_positive_aliases[value]
+            # TODO(voz): Decently common pattern, refactor at some point.
+            dup_guard = self._make_dupe_guard(stored_value)
+            if dup_guard:
+                stored_value = stored_value.add_guards(self.make_guards(dup_guard))
+            return stored_value


This should have been on #104921 - but I fat fingered the commit somewhere.

atalman · 2023-07-17T12:52:57Z

@voznesenskym Looks like there is consistent failure in our CI after this PR was merged:
https://hud.pytorch.org/hud/pytorch/pytorch/main/1?per_page=50&name_filter=nductor_torchbench_dynamic%2C%20

Looks like CUDA oom.

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 768.00 MiB. GPU 0 has a total capacty of 21.99 GiB of which 427.06 MiB is free. Process 17682 has 21.56 GiB memory in use. Of the allocated memory 20.96 GiB is allocated by PyTorch, and 282.55 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3194, in run
    ) = runner.load_model(
  File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 385, in load_model
    self.validate_model(model, example_inputs)
  File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1750, in validate_model
    raise NotImplementedError("Eager model failed to run") from e
NotImplementedError: Eager model failed to run

atalman · 2023-07-17T13:12:10Z

@pytorchbot revert -c ignoredsignal -m "Failing cuda11.8-py3.10-gcc7-sm86 / test (inductor_torchbench_dynamic) with CUDA oom"

pytorchmergebot · 2023-07-17T13:13:41Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2023-07-17T13:13:52Z

@voznesenskym your PR has been successfully reverted.

This reverts commit 82fb5ed. Reverted #103205 on behalf of https://github.com/atalman due to Failing cuda11.8-py3.10-gcc7-sm86 / test (inductor_torchbench_dynamic) with CUDA oom ([comment](#103205 (comment)))

voznesenskym · 2023-07-17T16:47:42Z

I wonder if we are failing because we graph break less now?

voznesenskym · 2023-07-17T17:07:15Z

I looked at the log, the oom is on another model, it seems?

2023-07-15T16:57:45.0635699Z cuda eval  llama                              
2023-07-15T16:58:22.4798235Z skipping cudagraphs due to input mutation
2023-07-15T16:58:22.8327903Z [2023-07-15 16:58:22,832] torch._dynamo.utils: [ERROR] Accuracy failed: allclose not within tol=0.0001
2023-07-15T16:58:22.8344321Z fail_accuracy

is the accuracy failure.

This reverts commit 94b3f9f. Fix [ghstack-poisoned]

This reverts commit 94b3f9f. Fix ghstack-source-id: 378a74aaaaafb0e468b55ad72d2cd9a9fea1a9c7 Pull Request resolved: #105345

…r improved graph breaks" This reverts commit 94b3f9f. Fix cc penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx ipiszy chenyang78 aakhundov [ghstack-poisoned]

This reverts commit 94b3f9f. Fix ghstack-source-id: c4d5ce1e2d64ac905d46aff0396b8349271aa49f Pull Request resolved: #105345 Fix

…graph breaks (#105345) This reverts commit 94b3f9f. Fix Pull Request resolved: #105345 Approved by: https://github.com/atalman

Set initial ghstack-source-id: aee0c9f4627d63000ba146006d494a2dbf7adcbf Pull Request resolved: #103205 wip WIP WIP WIP lint lint lint lint lint Fix Fix fixup fixup Fix

Set initial

2a93ea0

Set initial [ghstack-poisoned]

voznesenskym added a commit that referenced this pull request Jun 7, 2023

Set initial

8bb7ae7

Set initial ghstack-source-id: 79873ff77503d7488c3b2f3929560fb5d3d069aa Pull Request resolved: #103205

github-actions bot requested review from albanD, antoniojkim, bdhirsh, ezyang, jbschlosser, miladm, SherlockNoMad and wconstab June 7, 2023 21:36

github-actions bot added ciflow/inductor module: dynamo labels Jun 7, 2023

voznesenskym changed the title ~~Set initial~~ [WIP] SetVariable in dynamo Jun 7, 2023

voznesenskym commented Jun 7, 2023

View reviewed changes

torch/_dynamo/variables/builder.py Outdated Show resolved Hide resolved

wconstab reviewed Jun 7, 2023

View reviewed changes

torch/_dynamo/variables/lists.py Outdated Show resolved Hide resolved

wconstab reviewed Jun 7, 2023

View reviewed changes

torch/_dynamo/symbolic_convert.py Outdated Show resolved Hide resolved

ezyang requested a review from jansel June 8, 2023 17:10

voznesenskym commented Jun 8, 2023

View reviewed changes

jansel requested changes Jun 8, 2023

View reviewed changes

voznesenskym closed this Jun 9, 2023

Update on "[WIP] SetVariable in dynamo"

7d804f1

Set initial cc penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx ipiszy [ghstack-poisoned]

voznesenskym added a commit that referenced this pull request Jun 9, 2023

Set initial

949495b

Set initial ghstack-source-id: 030e4e6d7c5497c9f96ad9d36c94cdfeaf616761 Pull Request resolved: #103205 wip WIP WIP

Update on "[WIP] SetVariable in dynamo"

ee7f3df

Set initial cc penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx ipiszy [ghstack-poisoned]

voznesenskym added a commit that referenced this pull request Jun 9, 2023

Set initial

3da3456

Set initial ghstack-source-id: b7950036819956f2f0670b9ea278cda89d43bdff Pull Request resolved: #103205 wip WIP WIP WIP

voznesenskym reopened this Jun 9, 2023

voznesenskym changed the title ~~[WIP] SetVariable in dynamo~~ SetVariable in dynamo Jun 9, 2023

github-actions bot requested a review from wconstab June 9, 2023 18:46

voznesenskym requested a review from jansel June 9, 2023 18:46

Update on "SetVariable in dynamo"

7becadb

Set initial cc penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx ipiszy [ghstack-poisoned]

pytorchmergebot closed this in 82fb5ed Jul 15, 2023

voznesenskym commented Jul 15, 2023

View reviewed changes

pytorchmergebot added the Reverted label Jul 17, 2023

voznesenskym reopened this Jul 17, 2023

github-actions bot requested review from albanD, antoniojkim, bdhirsh, ezyang, jbschlosser, miladm, SherlockNoMad and wconstab July 17, 2023 17:34

voznesenskym added a commit that referenced this pull request Jul 17, 2023

Revert "Revert "SetVariable in dynamo (#103205)""

87b487a

This reverts commit 94b3f9f. Fix [ghstack-poisoned]

voznesenskym added a commit that referenced this pull request Jul 17, 2023

Revert "Revert "SetVariable in dynamo (#103205)""

1f96a6c

This reverts commit 94b3f9f. Fix ghstack-source-id: 378a74aaaaafb0e468b55ad72d2cd9a9fea1a9c7 Pull Request resolved: #105345

albanD removed their request for review July 17, 2023 19:24

voznesenskym added a commit that referenced this pull request Jul 17, 2023

Revert "Revert "SetVariable in dynamo (#103205)""

2bc87c2

This reverts commit 94b3f9f. Fix ghstack-source-id: c4d5ce1e2d64ac905d46aff0396b8349271aa49f Pull Request resolved: #105345 Fix

pytorchmergebot pushed a commit that referenced this pull request Jul 17, 2023

Revert "Revert "SetVariable in dynamo (#103205)"" + Fix for improved …

a6758cb

…graph breaks (#105345) This reverts commit 94b3f9f. Fix Pull Request resolved: #105345 Approved by: https://github.com/atalman

voznesenskym added a commit that referenced this pull request Jul 19, 2023

Set initial

293a2e6

Set initial ghstack-source-id: aee0c9f4627d63000ba146006d494a2dbf7adcbf Pull Request resolved: #103205 wip WIP WIP WIP lint lint lint lint lint Fix Fix fixup fixup Fix

voznesenskym added a commit that referenced this pull request Jul 21, 2023

Set initial

6c2e8bf

Set initial ghstack-source-id: aee0c9f4627d63000ba146006d494a2dbf7adcbf Pull Request resolved: #103205 wip WIP WIP WIP lint lint lint lint lint Fix Fix fixup fixup Fix

voznesenskym closed this Aug 7, 2023

voznesenskym added a commit that referenced this pull request Aug 7, 2023

Set initial

587d9ed

Set initial ghstack-source-id: aee0c9f4627d63000ba146006d494a2dbf7adcbf Pull Request resolved: #103205 wip WIP WIP WIP lint lint lint lint lint Fix Fix fixup fixup Fix

facebook-github-bot deleted the gh/voznesenskym/147/head branch August 11, 2023 14:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SetVariable in dynamo #103205

SetVariable in dynamo #103205

voznesenskym commented Jun 7, 2023 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jun 7, 2023 •

edited

Loading

voznesenskym Jun 8, 2023

voznesenskym Jul 15, 2023

atalman commented Jul 17, 2023

atalman commented Jul 17, 2023

pytorchmergebot commented Jul 17, 2023

pytorchmergebot commented Jul 17, 2023

voznesenskym commented Jul 17, 2023

voznesenskym commented Jul 17, 2023

SetVariable in dynamo #103205

SetVariable in dynamo #103205

Conversation

voznesenskym commented Jun 7, 2023 • edited by pytorch-bot bot Loading

pytorch-bot bot commented Jun 7, 2023 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/103205

❌ 1 New Failure, 1 Unrelated Failure

voznesenskym Jun 8, 2023

Choose a reason for hiding this comment

voznesenskym Jul 15, 2023

Choose a reason for hiding this comment

atalman commented Jul 17, 2023

atalman commented Jul 17, 2023

pytorchmergebot commented Jul 17, 2023

pytorchmergebot commented Jul 17, 2023

voznesenskym commented Jul 17, 2023

voznesenskym commented Jul 17, 2023

voznesenskym commented Jun 7, 2023 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jun 7, 2023 •

edited

Loading