Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RELAND] [CUDA graphs] Make CUDAGeneratorImpl capturable (ci-all edition) #48694

Closed
wants to merge 14 commits into from

Conversation

mcarilli
Copy link
Collaborator

@mcarilli mcarilli commented Dec 2, 2020

Resubmission of #47989 with attempted fix for the unexpected context creation that caused revert (#47989 (comment)).

Submitting from a ci-all branch because the failing test isn't public.

Diffs relative to master should be the same as #47989 's approved diffs, aside from the fix itself a5c80f6.

@dr-ci
Copy link

dr-ci bot commented Dec 2, 2020

💊 CI failures summary and remediations

As of commit 28d5a21 (more details on the Dr. CI page):


None of the CI failures appear to be your fault 💚



🚧 2 fixed upstream failures:

These were probably caused by upstream breakages that were already fixed.

Please rebase on the viable/strict branch (expand for instructions)

Since your merge base is older than viable/strict, run these commands:

git fetch https://github.com/pytorch/pytorch viable/strict
git rebase FETCH_HEAD

Check out the recency history of this "viable master" tracking branch.


This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 4 times.

Copy link
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ngimel has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@heitorschueroff heitorschueroff added module: cuda Related to torch.cuda, and CUDA support in general triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Dec 3, 2020
@facebook-github-bot
Copy link
Contributor

@ngimel merged this pull request in 31808dc.

@mcarilli mcarilli deleted the ci-all/cudagraphs_generator_diffs_v2 branch December 9, 2020 02:37
facebook-github-bot pushed a commit that referenced this pull request Dec 14, 2020
Summary:
Part 2 of #46148 refactor.  (part 1 was #48694.)
Contains
- a few more CUDAGeneratorImpl diffs to clean up graph capture interaction
- Capture and replay bindings that interact correctly with CUDAGeneratorImpl
- Tests.

Diffs compile and tests pass on my machine (ubuntu 20.04, cuda 11.0) but it needs finetuning for many CI builds.

See [Note [CUDA Graph-safe RNG states]](https://github.com/pytorch/pytorch/blob/02d89f9f1d7f32ebf7ec509d5c14b2f39690997a/aten/src/ATen/CUDAGeneratorImpl.h#L13-L85) for the strategy, based on #46148 (comment).

Pull Request resolved: #48875

Reviewed By: zou3519

Differential Revision: D25482654

Pulled By: ngimel

fbshipit-source-id: 634dbc4c6c9d7d0d9a62dc81a52d430561f905fe
@mcarilli mcarilli added the module: cuda graphs Ability to capture and then replay streams of CUDA kernels label Apr 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla signed Merged module: cuda graphs Ability to capture and then replay streams of CUDA kernels module: cuda Related to torch.cuda, and CUDA support in general open source triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants