Avoid reference invalidation in cuda SpectralOps' plan_caches #31861
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
peterbell10
commented
Jan 4, 2020
| @@ -261,9 +261,10 @@ static inline Tensor _run_cufft( | |||
| return output; | |||
| } | |||
|
|
|||
| // The cuFFT plan cache, defined in CuFFTUtils.h | |||
There was a problem hiding this comment.
This comment wasn't correct. plan_caches is not declared in CuFFTUtils.h and is only used in this file so made it static.
💊 CircleCI build failures summary and remediationsAs of commit 7769b5f:
Detailed failure analysisOne may explore the probable reasons each build failed interactively on the Dr. CI website. 🕵️ 1 new failure recognized by patternsThe following build failures do not appear to be due to upstream breakage:
|
ssnl
approved these changes
Jan 5, 2020
Could the crash reproducer be added as a slow test, maybe? Anyway, nice work! |
There was a problem hiding this comment.
@ezyang is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
wuhuikx
pushed a commit
to wuhuikx/pytorch
that referenced
this pull request
Jan 30, 2020
…h#31861) Summary: Fixes pytorch#31412 The root cause is `plan_caches` being resized in one thread while another holds a reference to an existing `CuFFTParamsLRUCache` which then becomes invalidated. I was able to reproduce the crash very reliably without this fix applied and no longer see it. Being a race condition, it's hard to say for sure though. Pull Request resolved: pytorch#31861 Differential Revision: D19312314 Pulled By: ezyang fbshipit-source-id: 06e4561128d503f2d70cdfe1982be0f3db2a8cf8
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #31412
The root cause is
plan_cachesbeing resized in one thread while another holds a reference to an existingCuFFTParamsLRUCachewhich then becomes invalidated.I was able to reproduce the crash very reliably without this fix applied and no longer see it. Being a race condition, it's hard to say for sure though.