Skip to content

[inductor] CI improvments #91283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 5 commits into from
Closed

[inductor] CI improvments #91283

wants to merge 5 commits into from

Conversation

@desertfire
Copy link
Contributor

@desertfire desertfire commented Dec 21, 2022 •
edited by pytorch-bot bot
Loading

Stack from ghstack (oldest at bottom):

Summary:

  1. Setting torch.backends.cudnn.deterministic to True helps to
    eliminate the eager_variance failures seen on CI
  2. Skip Triton failure instead of retry
  3. Some minor script cleanup is also included in this PR.

cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx

@pytorch-bot
Copy link

pytorch-bot bot commented Dec 21, 2022 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/91283

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 540110a:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@desertfire desertfire changed the title [WIP] investigate eager_variation [inductor] Set use_eval_mode for timm_model Dec 22, 2022
@desertfire
Update on "[inductor] Set use_eval_mode for timm_model"
0a9f675 
Summary: We need to set use_eval_mode when checking training accuracy,
not only for inductor but also for eager, to avoid randomness from
dropout. Some minor script clean up is also included in this PR.

[ghstack-poisoned]
@desertfire
Update on "[inductor] Set use_eval_mode for timm_model"
2173c1f 
Summary: We need to set use_eval_mode when checking training accuracy,
not only for inductor but also for eager, to avoid randomness from
dropout. Some minor script clean up is also included in this PR.

[ghstack-poisoned]
@desertfire desertfire added the topic: not user facing topic category label Dec 22, 2022
@ngimel
Copy link
Collaborator

ngimel commented Dec 22, 2022

cc @anijain2305. We can't set eval mode for timm models because that would disable bn testing, and we had serious bugs when we didn't test bn.
Instead, we should leave the mode as training, but set dropout to 0 to avoid non-determinism (also, my impression was that dropout is already 0 for most timm models by default)

@desertfire desertfire changed the title [inductor] Set use_eval_mode for timm_model [inductor] Set torch.backends.cudnn.deterministic Dec 22, 2022
@desertfire
Update on "[inductor] Set torch.backends.cudnn.deterministic"
d9db2f1 
Summary: Setting torch.backends.cudnn.deterministic to True helps to
eliminate the eager_variance failures seen on CI.
Some minor script clean up is also included in this PR.

[ghstack-poisoned]
@desertfire
Copy link
Contributor Author

cc @anijain2305. We can't set eval mode for timm models because that would disable bn testing, and we had serious bugs when we didn't test bn. Instead, we should leave the mode as training, but set dropout to 0 to avoid non-determinism (also, my impression was that dropout is already 0 for most timm models by default)

Discussed offline. Dropout is not the root cause here since it is already 0 for those failing models we have seen on CI. Setting torch.backends.cudnn.deterministic = True works, at least for the models I tried. Let's see if it solves all the eager_variance failures on CI.

@anijain2305
Copy link
Contributor

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 22, 2022
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: 2 additional jobs have failed, first few of them are: inductor ,inductor / cuda11.6-py3.10-gcc7-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)

Details for Dev Infra team Raised by workflow job

@desertfire desertfire changed the title [inductor] Set torch.backends.cudnn.deterministic [inductor] CI improvments Dec 22, 2022
@desertfire
Update on "[inductor] CI improvments"
540110a 
Summary:
1) Setting torch.backends.cudnn.deterministic to True helps to
eliminate the eager_variance failures seen on CI
2) Skip Triton failure instead of retry
3) Some minor script cleanup is also included in this PR.

[ghstack-poisoned]
desertfire added a commit that referenced this pull request Dec 22, 2022
@desertfire
[inductor] CI improvments
e9aba06 
Summary:
1) Setting torch.backends.cudnn.deterministic to True helps to
eliminate the eager_variance failures seen on CI
2) Skip Triton failure instead of retry
3) Some minor script cleanup is also included in this PR.

ghstack-source-id: 76e04e4
Pull Request resolved: #91283
@desertfire
Copy link
Contributor Author

@pytorchbot merge -f "Only affects inductor shards and they have already passed"

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@desertfire desertfire mentioned this pull request Jan 3, 2023
Closed
desertfire added a commit that referenced this pull request Jan 3, 2023
@desertfire
[inductor] Check for BackendCompilerFailed on CI
6bd7fc5 
Summary: #91283 skips certain
random triton failure on CI, but we need to check against the
BackendCompilerFailed exception type.

[ghstack-poisoned]
desertfire added a commit that referenced this pull request Jan 3, 2023
@desertfire
[inductor] Check for BackendCompilerFailed on CI
0322cfa 
Summary: #91283 skips certain
random triton failure on CI, but we need to check against the
BackendCompilerFailed exception type.

ghstack-source-id: 0db96f8
Pull Request resolved: #91634
pytorchmergebot pushed a commit that referenced this pull request Jan 3, 2023
@desertfire @pytorchmergebot
[inductor] Check for BackendCompilerFailed on CI (#91634)
6bf0e3b 
Summary: #91283 skips certain
random triton failure on CI, but we need to check against the
BackendCompilerFailed exception type.

Pull Request resolved: #91634
Approved by: https://github.com/ngimel
@facebook-github-bot facebook-github-bot deleted the gh/desertfire/56/head branch June 8, 2023 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anijain2305 anijain2305 anijain2305 approved these changes

@mlazos mlazos Awaiting requested review from mlazos

@ngimel ngimel Awaiting requested review from ngimel

@eellison eellison Awaiting requested review from eellison

Assignees

No one assigned

Labels

ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged module: dynamo topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@desertfire @ngimel @anijain2305 @pytorchmergebot