Set torch.backends.cudnn.enabled to false when testing accuracy #94363

desertfire · 2023-02-08T01:21:06Z

Stack from ghstack (oldest at bottom):

-> Set torch.backends.cudnn.enabled to false when testing accuracy #94363

Summary: It looks like setting torch.backends.cudnn.deterministic to
True is not enough for eliminating non-determinism when testing
benchmarks with --accuracy, so let's turn off cudnn completely.
With this change, mobilenet_v3_large does not show random failure on my
local environment. Also take this chance to clean up CI skip lists.

cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx

Summary: It looks like setting torch.backends.cudnn.deterministic to True is not enough for eliminating non-determinism when testing benchmarks with --accuracy, so let's turn off cudnn completely. With this change, mobilenet_v3_large does not show random failure on my local environment. Also take this chance to clean up CI skip lists. [ghstack-poisoned]

pytorch-bot · 2023-02-08T01:21:09Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/94363

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 2 Active SEVs

There are 2 currently active SEVs. If your PR is affected, please view them below:

❌ 3 Failures

As of commit e2cd8a6:

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following jobs failed but were present on the merge base 76ed1a8:

cuda11.7-py3.10-gcc7-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Summary: It looks like setting torch.backends.cudnn.deterministic to True is not enough for eliminating non-determinism when testing benchmarks with --accuracy, so let's turn off cudnn completely. With this change, mobilenet_v3_large does not show random failure on my local environment. Also take this chance to clean up CI skip lists. ghstack-source-id: c2c353c Pull Request resolved: #94363

benchmarks/dynamo/common.py

ezyang

OK, but cc @kurtamohler, it really should be deterministic...

…uracy" Summary: It looks like setting torch.backends.cudnn.deterministic to True is not enough for eliminating non-determinism when testing benchmarks with --accuracy, so let's turn off cudnn completely. With this change, mobilenet_v3_large does not show random failure on my local environment. Also take this chance to clean up CI skip lists. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

desertfire · 2023-02-08T16:50:35Z

Sigh, there is still inconsistency between local-run results and CI results.

desertfire · 2023-02-08T19:23:41Z

@pytorchbot merge

pytorchmergebot · 2023-02-08T19:25:47Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

huydhn · 2023-02-09T01:22:50Z

@pytorchbot revert -m 'This change fails in trunk https://hud.pytorch.org/pytorch/pytorch/commit/7bfc59993d25c444eccb6cd77e85e4dd0a348b7e running out of memory. Mark this as weird because it was green in PR' -c weird

pytorchmergebot · 2023-02-09T01:24:28Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2023-02-09T01:24:40Z

@desertfire your PR has been successfully reverted.

…cy (#94363)" This reverts commit 7bfc599. Reverted #94363 on behalf of https://github.com/huydhn due to This change fails in trunk https://hud.pytorch.org/pytorch/pytorch/commit/7bfc59993d25c444eccb6cd77e85e4dd0a348b7e running out of memory. Mark this as weird because it was green in PR

desertfire · 2023-02-09T01:38:02Z

@pytorchbot rebase

pytorchmergebot · 2023-02-10T14:12:44Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2023-02-10T14:12:51Z

Rebase failed due to

Aborting rebase because rebasing the branch resulted in the same sha as the target branch.
This usually happens because the PR has already been merged.  Please rebase locally and push.

Raised by https://github.com/pytorch/pytorch/actions/runs/4144664499

Summary: This is the reland of #94363. [ghstack-poisoned]

Summary: This is the reland of #94363. ghstack-source-id: c859ff1 Pull Request resolved: #94712

…to false when testing accuracy" Summary: This is the reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

…sting accuracy" Summary: This is the reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

…to false when testing accuracy" Summary: This is the reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

…sting accuracy" Summary: This is the reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

Summary: This is the reland of #94363. ghstack-source-id: 2837cc4 Pull Request resolved: #94712

…to false when testing accuracy" Summary: This is the reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

Summary: This is the reland of #94363. ghstack-source-id: f871094 Pull Request resolved: #94712

…sting accuracy" Summary: This is the reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

Summary: This is the reland of #94363. ghstack-source-id: 0308943 Pull Request resolved: #94712

…to false when testing accuracy" Summary: This is the reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

…sting accuracy" Summary: This is the reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

…to false when testing accuracy" Summary: This is a reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

Summary: This is the reland of #94363. ghstack-source-id: 4c1d48a Pull Request resolved: #94712

…sting accuracy" Summary: This is a reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

…to false when testing accuracy" Summary: This is a reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

…sting accuracy" Summary: This is a reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

Summary: This is the reland of #94363. ghstack-source-id: aa4d986 Pull Request resolved: #94712

…to false when testing accuracy" Summary: This is a reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

Summary: This is the reland of #94363. ghstack-source-id: 218ebd8 Pull Request resolved: #94712

…sting accuracy" Summary: This is a reland of #94363. cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

github-actions bot added ciflow/inductor module: dynamo labels Feb 8, 2023

desertfire added the ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR label Feb 8, 2023

desertfire requested review from ezyang, ngimel and williamwen42 February 8, 2023 01:24

desertfire added the topic: not user facing topic category label Feb 8, 2023

ezyang reviewed Feb 8, 2023

View reviewed changes

benchmarks/dynamo/common.py Outdated Show resolved Hide resolved

ezyang approved these changes Feb 8, 2023

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Feb 8, 2023

pytorchmergebot added the Merged label Feb 8, 2023

pytorchmergebot closed this in 7bfc599 Feb 8, 2023

huydhn reopened this Feb 9, 2023

pytorchmergebot added the Reverted label Feb 9, 2023

desertfire removed ciflow/trunk Trigger trunk jobs on your pull request ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR labels Feb 10, 2023

desertfire closed this Feb 10, 2023

desertfire mentioned this pull request Feb 12, 2023

[Reland] Set torch.backends.cudnn.enabled to false when testing accuracy #94712

Closed

desertfire added a commit that referenced this pull request Feb 12, 2023

[Reland] Set torch.backends.cudnn.enabled to false when testing accuracy

de0aed9

Summary: This is the reland of #94363. [ghstack-poisoned]

desertfire added a commit that referenced this pull request Feb 12, 2023

[Reland] Set torch.backends.cudnn.enabled to false when testing accuracy

ad1dbff

Summary: This is the reland of #94363. ghstack-source-id: c859ff1 Pull Request resolved: #94712

desertfire added a commit that referenced this pull request Feb 13, 2023

[Reland] Set torch.backends.cudnn.enabled to false when testing accuracy

aa3821d

Summary: This is the reland of #94363. ghstack-source-id: 2837cc4 Pull Request resolved: #94712

pytorchmergebot pushed a commit that referenced this pull request Feb 14, 2023

[Reland] Set torch.backends.cudnn.enabled to false when testing accuracy

a0aa8b3

Summary: This is the reland of #94363. ghstack-source-id: f871094 Pull Request resolved: #94712

desertfire added a commit that referenced this pull request Feb 15, 2023

[Reland] Set torch.backends.cudnn.enabled to false when testing accuracy

6e69cbd

Summary: This is the reland of #94363. ghstack-source-id: 0308943 Pull Request resolved: #94712

pytorchmergebot pushed a commit that referenced this pull request Feb 16, 2023

[Reland] Set torch.backends.cudnn.enabled to false when testing accuracy

6b928b7

Summary: This is the reland of #94363. ghstack-source-id: 4c1d48a Pull Request resolved: #94712

pytorchmergebot pushed a commit that referenced this pull request Feb 16, 2023

[Reland] Set torch.backends.cudnn.enabled to false when testing accuracy

81e2eee

Summary: This is the reland of #94363. ghstack-source-id: aa4d986 Pull Request resolved: #94712

desertfire added a commit that referenced this pull request Feb 17, 2023

[Reland] Set torch.backends.cudnn.enabled to false when testing accuracy

b011764

Summary: This is the reland of #94363. ghstack-source-id: 218ebd8 Pull Request resolved: #94712

facebook-github-bot deleted the gh/desertfire/65/head branch June 8, 2023 16:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Set torch.backends.cudnn.enabled to false when testing accuracy #94363

Set torch.backends.cudnn.enabled to false when testing accuracy #94363

Uh oh!

desertfire commented Feb 8, 2023 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Feb 8, 2023 •

edited

Loading

Uh oh!

Uh oh!

ezyang left a comment

Uh oh!

desertfire commented Feb 8, 2023

Uh oh!

desertfire commented Feb 8, 2023

Uh oh!

pytorchmergebot commented Feb 8, 2023

Uh oh!

huydhn commented Feb 9, 2023

Uh oh!

pytorchmergebot commented Feb 9, 2023

Uh oh!

pytorchmergebot commented Feb 9, 2023

Uh oh!

desertfire commented Feb 9, 2023

Uh oh!

pytorchmergebot commented Feb 10, 2023

Uh oh!

pytorchmergebot commented Feb 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Set torch.backends.cudnn.enabled to false when testing accuracy #94363

Set torch.backends.cudnn.enabled to false when testing accuracy #94363

Uh oh!

Conversation

desertfire commented Feb 8, 2023 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Feb 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/94363

❗ 2 Active SEVs

❌ 3 Failures

Uh oh!

Uh oh!

ezyang left a comment

Choose a reason for hiding this comment

Uh oh!

desertfire commented Feb 8, 2023

Uh oh!

desertfire commented Feb 8, 2023

Uh oh!

pytorchmergebot commented Feb 8, 2023

Merge started

Uh oh!

huydhn commented Feb 9, 2023

Uh oh!

pytorchmergebot commented Feb 9, 2023

Uh oh!

pytorchmergebot commented Feb 9, 2023

Uh oh!

desertfire commented Feb 9, 2023

Uh oh!

pytorchmergebot commented Feb 10, 2023

Uh oh!

pytorchmergebot commented Feb 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

desertfire commented Feb 8, 2023 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Feb 8, 2023 •

edited

Loading