Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix MemoryAsync to keep a weakref to stream #5264

Merged
merged 4 commits into from Jun 2, 2021
Merged

Fix MemoryAsync to keep a weakref to stream #5264

merged 4 commits into from Jun 2, 2021

Conversation

kmaehashi
Copy link
Member

@kmaehashi kmaehashi commented May 26, 2021 •
edited

Currently MemoryAsync is holding a pointer to the stream, but this is unsafe because:

  • In general it is prohibited to pass destoryed ptr to CUDA APIs (I sometimes experienced SegV).
  • The stream ptr may be reused. (e.g., cudaStreamCreate(&x); cudaStreamDestroy(x); cudaStreamCreate(&y) // x != y is not guaranteed)

ref: #4592

@kmaehashi
Copy link
Member Author

pfnCI, test this please.

@kmaehashi
Copy link
Member Author

cc/ @leofang

@kmaehashi kmaehashi mentioned this pull request May 26, 2021
Merged
@kmaehashi kmaehashi requested a review from leofang May 26, 2021 04:37
@kmaehashi kmaehashi added cat:bug Bugs prio:high to-be-backported Pull-requests to be backported to stable branch no-compat Disrupts backward compatibility labels May 26, 2021
@chainer-ci
Copy link
Member

Jenkins CI test (for commit cd88bce, target branch master) succeeded!

leofang
leofang previously approved these changes May 26, 2021
View reviewed changes
Copy link
Member

@leofang leofang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just left a nitpick.

It is weird that I cannot reproduce any errors or segfaults on my side... I thought the ok_errors are enough to handle the stream lifetime issue 🤔

cupy/cuda/memory.pyx Outdated Show resolved Hide resolved
@leofang
Copy link
Member

leofang commented May 26, 2021

Jenkins, test this please

@kmaehashi @leofang
Update cupy/cuda/memory.pyx
769d4e5 
Co-authored-by: Leo Fang <leofang@bnl.gov>
@kmaehashi
Copy link
Member Author

pfnCI, test this please.

@chainer-ci
Copy link
Member

Jenkins CI test (for commit 769d4e5, target branch master) succeeded!

@emcastillo emcastillo assigned takagi and unassigned emcastillo May 28, 2021
@takagi takagi added this to the v10.0.0a2 milestone Jun 2, 2021
@takagi takagi merged commit 29eb596 into cupy:master Jun 2, 2021
chainer-ci pushed a commit to chainer-ci/cupy that referenced this pull request Jun 2, 2021
@takagi @chainer-ci
Merge pull request cupy#5264 from kmaehashi/fix-async-mem-stream
a18da1e 
Fix MemoryAsync to keep a weakref to stream
@chainer-ci chainer-ci mentioned this pull request Jun 2, 2021
Merged
@kmaehashi kmaehashi deleted the fix-async-mem-stream branch June 2, 2021 06:06
@kmaehashi kmaehashi mentioned this pull request Jun 2, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@takagi takagi takagi approved these changes

@leofang leofang leofang left review comments

Assignees

@takagi takagi

Labels
cat:bug Bugs no-compat Disrupts backward compatibility prio:high to-be-backported Pull-requests to be backported to stable branch
Projects
None yet
Milestone
v10.0.0a2
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@kmaehashi @chainer-ci @leofang @takagi @emcastillo