Always inspect the task-local context when verifying before freeing. #1462
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I assume the same test of distributed training would be sufficient to test this? |
|
Yes, that would suffice. |
For some reason, the thread-bound context can become desynchronized from the task-local one. Generally we don't notice this, because we prefix every API call with a call that synchronizes both. Here, however, we explicitly didn't to avoid initializing the state as that was thought to cause the kind of initialization that may have to yield (which is unsupported when done so from a finalizer). However, just creating the task local state shouldn't result in yield, only creating a stream does, like querying `active_state` as was done before #1383.
Codecov Report
@@ Coverage Diff @@
## master #1462 +/- ##
==========================================
- Coverage 72.64% 72.63% -0.02%
==========================================
Files 131 131
Lines 9805 9805
==========================================
- Hits 7123 7122 -1
- Misses 2682 2683 +1
Continue to review full report at Codecov.
|
|
Awaiting a response, let's go ahead and merge this since it does fix a known issue. |
simonbyrne
pushed a commit
to simonbyrne/CUDA.jl
that referenced
this pull request
Nov 13, 2023
…uliaGPU#1462) For some reason, the thread-bound context can become desynchronized from the task-local one. Generally we don't notice this, because we prefix every API call with a call that synchronizes both. Here, however, we explicitly didn't to avoid initializing the state as that was thought to cause the kind of initialization that may have to yield (which is unsupported when done so from a finalizer). However, just creating the task local state shouldn't result in yield, only creating a stream does, like querying `active_state` as was done before JuliaGPU#1383.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For some reason, the thread-bound context can become desynchronized
from the task-local one. Generally we don't notice this, because we
prefix every API call with a call that synchronizes both. Here, however,
we explicitly didn't to avoid initializing the state as that was thought
to cause the kind of initialization that may have to yield (which is
unsupported when done so from a finalizer).
However, just creating the task local state shouldn't result in yield,
only creating a stream does, like querying
active_stateas was donebefore #1383.
Fixes #1454.
@DhairyaLGandhi Could you verify this doesn't re-introduce the errors when finalizing arrays?
Ultimately though this needs JuliaLang/julia#35689.