Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[compiled autograd] clear compiled_autograd_verbose once test is done #126148

Closed
wants to merge 5 commits into from
Closed

[compiled autograd] clear compiled_autograd_verbose once test is done #126148

wants to merge 5 commits into from

Conversation

xmfan
Copy link
Member

@xmfan xmfan commented May 14, 2024 •
edited

This was referenced May 14, 2024
Closed
Closed
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented May 14, 2024 •
edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126148

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (6 Unrelated Failures)

As of commit 0bc1814 with merge base 91bf952 (image):

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

xmfan added a commit that referenced this pull request May 14, 2024
@xmfan
[compiled autograd] clear compiled_autograd_verbose once test is done
95b8004 
ghstack-source-id: 7a16e9d6694e76ff82359c8345711b7b177ab5ee
Pull Request resolved: #126148
@xmfan
Update on "[compiled autograd] clear compiled_autograd_verbose once t…
07b9ec2 
…est is done"


verbose flag leaks into tests ran after

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang

[ghstack-poisoned]
xmfan added a commit that referenced this pull request May 14, 2024
@xmfan
[compiled autograd] clear compiled_autograd_verbose once test is done
463913e 
ghstack-source-id: f817bd618a06b97a86ef1262dd457cf19879c548
Pull Request resolved: #126148
@xmfan xmfan marked this pull request as ready for review May 14, 2024 15:31
@xmfan xmfan requested a review from jansel May 14, 2024 15:33
@xmfan
Copy link
Member Author

xmfan commented May 15, 2024

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 15, 2024
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: Approvers from one of the following sets are needed:

  • superuser (pytorch/metamates)
  • Core Reviewers (mruberry, lezcano, Skylion007, ngimel, peterbell10)
  • Core Maintainers (soumith, gchanan, ezyang, dzhulgakov, malfet)
Details for Dev Infra team Raised by workflow job

Failing merge rule: Core Maintainers

@xmfan
Copy link
Member Author

xmfan commented May 16, 2024

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x 643f57a78248d13d941da8829541cb515ede4422 returned non-zero exit code 1

Auto-merging test/dynamo/test_repros.py
CONFLICT (content): Merge conflict in test/dynamo/test_repros.py
Auto-merging torch/_inductor/utils.py
error: could not apply 643f57a7824... [inductor] Clear cache on ctx manager exit
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
Details for Dev Infra team Raised by workflow job

@xmfan
Update on "[compiled autograd] clear compiled_autograd_verbose once t…
483b428 
…est is done"


verbose flag leaks into tests ran after

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang

[ghstack-poisoned]
xmfan added a commit that referenced this pull request May 16, 2024
@xmfan
[compiled autograd] clear compiled_autograd_verbose once test is done
0272611 
ghstack-source-id: 03f335799def42f89f37eda70dcdb1cbd4656c62
Pull Request resolved: #126148
@xmfan
Copy link
Member Author

xmfan commented May 16, 2024

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: 1 jobs have failed, first few of them are: inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)

Details for Dev Infra team Raised by workflow job

@xmfan
Update on "[compiled autograd] clear compiled_autograd_verbose once t…
2fca582 
…est is done"


verbose flag leaks into tests ran after

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang

[ghstack-poisoned]
xmfan added a commit that referenced this pull request May 16, 2024
@xmfan
[compiled autograd] clear compiled_autograd_verbose once test is done
f87a6fe 
ghstack-source-id: 03f335799def42f89f37eda70dcdb1cbd4656c62
Pull Request resolved: #126148
@xmfan
Copy link
Member Author

xmfan commented May 16, 2024

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

Dig deeper by viewing the failures on hud

Details for Dev Infra team Raised by workflow job

Failing merge rule: Core Maintainers

@xmfan
Update
0bc1814 
[ghstack-poisoned]
@xmfan xmfan mentioned this pull request May 16, 2024
Closed
@xmfan
Copy link
Member Author

xmfan commented May 16, 2024

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

bilal2vec pushed a commit to bilal2vec/pytorch that referenced this pull request May 16, 2024
@xmfan @pytorchmergebot
[compiled autograd] clear compiled_autograd_verbose once test is done (…
93524cf 
…pytorch#126148)

verbose flag leaks into tests ran after

Pull Request resolved: pytorch#126148
Approved by: https://github.com/jansel
ghstack dependencies: pytorch#126144, pytorch#126146
@xmfan xmfan mentioned this pull request May 17, 2024
Closed
@xmfan xmfan closed this May 17, 2024
@xmfan xmfan mentioned this pull request May 18, 2024
Closed
ZelboK pushed a commit to ZelboK/pytorch that referenced this pull request May 19, 2024
@xmfan @ZelboK
[compiled autograd] clear compiled_autograd_verbose once test is done (…
b29fd1f 
…pytorch#126148)

verbose flag leaks into tests ran after

Pull Request resolved: pytorch#126148
Approved by: https://github.com/jansel
ghstack dependencies: pytorch#126144, pytorch#126146
pytorchmergebot pushed a commit that referenced this pull request May 19, 2024
@xmfan @pytorchmergebot
[compiled autograd] log in cpp using python logger (#126483)
be67985 
Internal infra may not preserve python and c++ log ordering e.g. MAST logs: https://fburl.com/mlhub/38576cxn, all the `[python_compiled_autograd.cpp] Creating cache entry [...]` logs of the entire run are at the beginning of the file

Pull Request resolved: #126483
Approved by: https://github.com/jansel
ghstack dependencies: #126144, #126146, #126148
pytorchmergebot pushed a commit that referenced this pull request May 19, 2024
@xmfan @pytorchmergebot
[compiled autograd] Better cache miss logging (#126602)
5fb11cd 
- log only first node key cache miss
- log existing node key sizes
- log which node's collected sizes became dynamic
e.g.
```
DEBUG:torch._dynamo.compiled_autograd.__compiled_autograd_verbose:Cache miss due to new autograd node: torch::autograd::GraphRoot (NodeCall 0) with key size 39, previous key sizes=[]
...
DEBUG:torch._dynamo.compiled_autograd.__compiled_autograd_verbose:Cache miss due to new autograd node: torch::autograd::AccumulateGrad (NodeCall 5) with key size 32, previous key sizes=[21]
...
DEBUG:torch._dynamo.compiled_autograd.__compiled_autograd_verbose:Cache miss due to dynamic shapes: collected size idx 0 of torch::autograd::GraphRoot (NodeCall 0)
DEBUG:torch._dynamo.compiled_autograd.__compiled_autograd_verbose:Cache miss due to dynamic shapes: collected size idx 2 of SumBackward0 (NodeCall 1)
DEBUG:torch._dynamo.compiled_autograd.__compiled_autograd_verbose:Cache miss due to dynamic shapes: collected size idx 4 of SumBackward0 (NodeCall 1)
DEBUG:torch._dynamo.compiled_autograd.__compiled_autograd_verbose:Cache miss due to dynamic shapes: collected size idx 2 of ReluBackward0 (NodeCall 2)
DEBUG:torch._dynamo.compiled_autograd.__compiled_autograd_verbose:Cache miss due to dynamic shapes: collected size idx 9 of AddmmBackward0 (NodeCall 3)
DEBUG:torch._dynamo.compiled_autograd.__compiled_autograd_verbose:Cache miss due to dynamic shapes: collected size idx 2 of torch::autograd::AccumulateGrad (NodeCall 5)
DEBUG:torch._dynamo.compiled_autograd.__compiled_autograd_verbose:Cache miss due to dynamic shapes: collected size idx 2 of ReluBackward0 (NodeCall 6)
```

Pull Request resolved: #126602
Approved by: https://github.com/jansel
ghstack dependencies: #126144, #126146, #126148, #126483
@github-actions github-actions bot deleted the gh/xmfan/51/head branch June 16, 2024 02:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@peterbell10 peterbell10 peterbell10 left review comments

@jansel jansel jansel approved these changes

Assignees
No one assigned
Labels
ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged module: dynamo module: inductor topic: not user facing topic category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@xmfan @pytorchmergebot @jansel @peterbell10