[CUDA Graphs] Add option to dump a captured graph for debugging #85519

eqy · 2022-09-22T23:52:38Z

CC @xwang233 @ptrblck @ngimel

pytorch-bot · 2022-09-22T23:52:41Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/85519

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit c08266d:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ptrblck · 2022-09-29T18:22:59Z

aten/src/ATen/cuda/CUDAGraph.cpp

+    AT_CUDA_CHECK(cudaGraphDestroy(graph_));
+    has_graph_ = false;
+  } else {
+    TORCH_WARN("DEBUG: TORCH_CUDAGRAPHS_DEBUG_PATH detected. graph_ will not be freed");


I think this is outdated.

ptrblck · 2022-09-29T19:01:16Z

torch/cuda/graphs.py

+    def debug_dump(self):
+        r"""
+        Calls a debugging function to dump the graph if the dump path has been
+        set via torch._C._cuda_setCudaGraphsDebugPath()


Should it be torch._C._cuda_set_graphs_debug_path?

eqy · 2022-09-29T23:51:51Z

@pytorchmergebot rebase

pytorchmergebot · 2022-09-29T23:53:54Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2022-09-29T23:54:00Z

Successfully rebased cuda_graphs_debugging onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cuda_graphs_debugging && git pull --rebase)

ngimel · 2022-09-30T00:24:26Z

aten/src/ATen/cuda/CUDAGraph.cpp

+    TORCH_WARN("DEBUG: calling debug_dump()");
+    if (has_graph_) {
+      TORCH_WARN("DEBUG: calling cudaGraphDebugDotPrint() with ", _cuda_graphs_debug_path);
+      C10_CUDA_CHECK_WARN(cudaGraphDebugDotPrint(graph_, _cuda_graphs_debug_path.c_str(), 1<<10)); // most verbose output


you can destroy graph_ now?

ngimel · 2022-09-30T00:33:24Z

torch/csrc/cuda/Module.cpp

@@ -979,6 +984,10 @@ static struct PyMethodDef _THCPModule_methods[] = {
     THCPModule_setBenchmarkLimitCuDNN,
     METH_O,
     nullptr},
+    {"_cuda_set_graphs_debug_path",


I think instead of _cuda_set_graphs_debug_path you should have something like enable_cudagraph_debug_mode that will preserve graph_ object for further inspection - you might want to do something other than dumping dot graph. As for path to dump the file, debug_dump itself should accept path argument - that would make it more consistent with other io functions.

As for path to dump the file, debug_dump itself should accept path argument - that would make it more consistent with other io functions.

👍 as this was also my first approach while testing the PR without looking at the code

facebook-github-bot · 2022-10-04T00:15:01Z

/easycla

As part of the transition to the PyTorch Foundation, this project now requires contributions be covered under the new CLA. See #85559 for additional details.

This comment will trigger a new check of this PR. If you are already covered, you will simply see a new "EasyCLA" check that passes. If you are not covered, a bot will leave a new comment with a link to sign.

linux-foundation-easycla · 2022-10-04T00:15:05Z

The committers listed above are authorized under a signed CLA.

✅ login: eqy (004a71e, e86a848, 30bc226, 8c90fd6, af0c3cc, 2cc6cef)

eqy · 2022-11-03T04:27:18Z

@pytorchmergebot rebase

pytorchmergebot · 2022-11-03T04:29:03Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2022-11-03T04:29:07Z

Successfully rebased cuda_graphs_debugging onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cuda_graphs_debugging && git pull --rebase)

eqy · 2022-11-22T18:37:22Z

@pytorchmergebot rebase

pytorchmergebot · 2022-11-22T18:39:22Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2022-11-22T18:39:25Z

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch rebase refs/remotes/origin/viable/strict pull/85519/head returned non-zero exit code 1

Rebasing (1/12)
Auto-merging torch/csrc/cuda/Graph.cpp
CONFLICT (content): Merge conflict in torch/csrc/cuda/Graph.cpp
error: could not apply ca2cc45aad... check in
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply ca2cc45aad... check in

Raised by https://github.com/pytorch/pytorch/actions/runs/3526151366

eqy · 2022-11-25T03:38:40Z

@pytorchmergebot rebase

pytorchmergebot · 2022-11-25T03:40:22Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2022-11-25T03:40:26Z

Successfully rebased cuda_graphs_debugging onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cuda_graphs_debugging && git pull --rebase)

eqy · 2022-12-06T18:44:15Z

@pytorchmergebot rebase

pytorchmergebot · 2022-12-06T18:46:53Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2022-12-06T18:47:02Z

Successfully rebased cuda_graphs_debugging onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cuda_graphs_debugging && git pull --rebase)

eqy · 2022-12-06T22:01:22Z

@pytorchmergebot merge

pytorchmergebot · 2022-12-06T22:03:01Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

@xwang233

…rch#85519) CC @xwang233 @ptrblck @ngimel Pull Request resolved: pytorch#85519 Approved by: https://github.com/ngimel

facebook-github-bot added the cla signed label Sep 23, 2022

pytorchbot added the open source label Sep 23, 2022

eqy added the ciflow/trunk Trigger trunk jobs on your pull request label Sep 28, 2022

eqy marked this pull request as ready for review September 28, 2022 22:06

ptrblck reviewed Sep 29, 2022

View reviewed changes

pytorchmergebot force-pushed the cuda_graphs_debugging branch from c2575ef to 7f0fba6 Compare September 29, 2022 23:54

ngimel reviewed Sep 30, 2022

View reviewed changes

ngimel mentioned this pull request Oct 18, 2022

New APIs for cuda graph inspection and manipulation #87163

Open

eqy force-pushed the cuda_graphs_debugging branch from 7f0fba6 to 2cc6cef Compare October 28, 2022 18:57

eqy changed the title ~~[WIP][CUDA Graphs] Add option to dump a captured graph for debugging~~ [CUDA Graphs] Add option to dump a captured graph for debugging Oct 28, 2022

soulitzer added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Oct 31, 2022

pytorchmergebot force-pushed the cuda_graphs_debugging branch from fe5ea40 to bc4c71f Compare November 3, 2022 04:29

ngimel approved these changes Nov 22, 2022

View reviewed changes

eqy force-pushed the cuda_graphs_debugging branch from 1b31a5d to 3b1a8e5 Compare November 22, 2022 19:15

pytorchmergebot force-pushed the cuda_graphs_debugging branch from bb5056d to d5784ab Compare November 25, 2022 03:40

eqy force-pushed the cuda_graphs_debugging branch from d5784ab to a39a252 Compare December 3, 2022 00:42

eqy added 11 commits December 6, 2022 18:46

check in

09b1f09

add missing declaration

e3756d2

guards for ROCM

3362274

fix

cc2da9e

update api

7f2a779

Update CUDAGraph.cpp

1848fa4

lint

f085412

Update Graph.cpp

ddafae9

Update Graph.cpp

09c10a1

fix

be8f552

lint

c08266d

pytorchmergebot force-pushed the cuda_graphs_debugging branch from a39a252 to c08266d Compare December 6, 2022 18:47

pytorchmergebot added the Merged label Dec 6, 2022

pytorchmergebot closed this in 62e450d Dec 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CUDA Graphs] Add option to dump a captured graph for debugging #85519

[CUDA Graphs] Add option to dump a captured graph for debugging #85519

eqy commented Sep 22, 2022

pytorch-bot bot commented Sep 22, 2022 •

edited

ptrblck Sep 29, 2022

ptrblck Sep 29, 2022

eqy commented Sep 29, 2022

pytorchmergebot commented Sep 29, 2022

pytorchmergebot commented Sep 29, 2022

ngimel Sep 30, 2022

ngimel Sep 30, 2022

ptrblck Sep 30, 2022

facebook-github-bot commented Oct 4, 2022

linux-foundation-easycla bot commented Oct 4, 2022 •

edited

eqy commented Nov 3, 2022

pytorchmergebot commented Nov 3, 2022

pytorchmergebot commented Nov 3, 2022

eqy commented Nov 22, 2022

pytorchmergebot commented Nov 22, 2022

pytorchmergebot commented Nov 22, 2022

eqy commented Nov 25, 2022

pytorchmergebot commented Nov 25, 2022

pytorchmergebot commented Nov 25, 2022

eqy commented Dec 6, 2022

pytorchmergebot commented Dec 6, 2022

pytorchmergebot commented Dec 6, 2022

eqy commented Dec 6, 2022

pytorchmergebot commented Dec 6, 2022

[CUDA Graphs] Add option to dump a captured graph for debugging #85519

[CUDA Graphs] Add option to dump a captured graph for debugging #85519

Conversation

eqy commented Sep 22, 2022

pytorch-bot bot commented Sep 22, 2022 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/85519

✅ No Failures

ptrblck Sep 29, 2022

Choose a reason for hiding this comment

ptrblck Sep 29, 2022

Choose a reason for hiding this comment

eqy commented Sep 29, 2022

pytorchmergebot commented Sep 29, 2022

pytorchmergebot commented Sep 29, 2022

ngimel Sep 30, 2022

Choose a reason for hiding this comment

ngimel Sep 30, 2022

Choose a reason for hiding this comment

ptrblck Sep 30, 2022

Choose a reason for hiding this comment

facebook-github-bot commented Oct 4, 2022

linux-foundation-easycla bot commented Oct 4, 2022 • edited

eqy commented Nov 3, 2022

pytorchmergebot commented Nov 3, 2022

pytorchmergebot commented Nov 3, 2022

eqy commented Nov 22, 2022

pytorchmergebot commented Nov 22, 2022

pytorchmergebot commented Nov 22, 2022

eqy commented Nov 25, 2022

pytorchmergebot commented Nov 25, 2022

pytorchmergebot commented Nov 25, 2022

eqy commented Dec 6, 2022

pytorchmergebot commented Dec 6, 2022

pytorchmergebot commented Dec 6, 2022

eqy commented Dec 6, 2022

pytorchmergebot commented Dec 6, 2022

Merge started

pytorch-bot bot commented Sep 22, 2022 •

edited

linux-foundation-easycla bot commented Oct 4, 2022 •

edited