Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix cudnn RNN reproducibility problem #90522

Closed
wants to merge 1 commit into from
Closed

fix cudnn RNN reproducibility problem #90522

wants to merge 1 commit into from

Conversation

cyyever
Copy link
Collaborator

@cyyever cyyever commented Dec 9, 2022 •
edited

Fixes #74177

Since RNN code use static variables to cache state, we store an atomic_flag in RNG generator to notify new seed changes and generate new random state for RNN. The additional cost is that the it must check the atomic_flag each time to ensure reproducibility. This may be ugly but it is the best way currently without large code refactoring

@pytorch-bot
Copy link

pytorch-bot bot commented Dec 9, 2022 •
edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/90522

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit ee38dce:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@soulitzer soulitzer requested a review from ngimel December 9, 2022 17:08
@soulitzer soulitzer added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Dec 9, 2022
@pytorch-bot
Copy link

pytorch-bot bot commented Dec 11, 2022

You are not authorized to force merges to this repository. Please use the regular @pytorchmergebot merge command instead

@cyyever
Copy link
Collaborator Author

cyyever commented Dec 11, 2022

@pytorchmergebot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 11, 2022
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: The following mandatory check(s) failed (Rule superuser):

Dig deeper by viewing the failures on hud

Details for Dev Infra team Raised by workflow job

@cyyever
Copy link
Collaborator Author

cyyever commented Dec 15, 2022

@pytorchmergebot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@cyyever cyyever deleted the rnnfix branch December 15, 2022 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ngimel ngimel ngimel approved these changes

Assignees
No one assigned
Labels
ciflow/trunk Trigger trunk jobs on your pull request Merged open source triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

get_dropout_state, seed and reproducibility
5 participants
@cyyever @pytorchmergebot @ngimel @soulitzer @pytorchbot