[Pytorch Edge] Support profiling kineto events from external source #64397

kimishpatel · 2021-09-01T21:24:15Z

Stack from ghstack:

-> [Pytorch Edge] Support profiling kineto events from external source #64397

Summary:
This diff exposes a way to add events to kineto profiler from external
source.
This can be a backend that executes a subgraph and wants to record this
execution in kineto profiler.
This diff also adds "backend" metadata to identify the backend an event
would have executed on.

Test Plan:
test_lite_interpreter

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: D30710710

Here is a chrome trace output for example delegated backend

Summary: This diff exposes a way to add events to kineto profiler from external source. This can be a backend that executes a subgraph and wants to record this execution in kineto profiler. This diff also adds "backend" metadata to identify the backend an event would have executed on. Test Plan: test_lite_interpreter Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

facebook-github-bot · 2021-09-01T21:24:21Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/64397
📄 Preview docs built from this PR
📄 Preview C++ docs built from this PR
↩️ [fb-only] Re-run with SSH instructions
🔧 Opt-in to CIFlow to control what jobs run on your PRs

💊 CI failures summary and remediations

As of commit 3a28ee8 (more details on the Dr. CI page):

1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

linux-xenial-py3.6-gcc7-bazel-test / build-and-test (1/1)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2021-10-07T22:54:20.9448601Z collect2: error: ld returned 1 exit status

2021-10-07T22:54:20.9435737Z �[1A�[K
2021-10-07T22:54:20.9436053Z �[1A�[K
2021-10-07T22:54:20.9436349Z �[1A�[K
2021-10-07T22:54:20.9436660Z �[1A�[K
2021-10-07T22:54:20.9436962Z �[1A�[K
2021-10-07T22:54:20.9437272Z �[1A�[K
2021-10-07T22:54:20.9437576Z �[1A�[K
2021-10-07T22:54:20.9439859Z �[1A�[Kbazel-out/k8-fastbuild/bin/_objs/jit_tests/test_backend_compiler_lib.pic.o:test_backend_compiler_lib.cpp:function torch::jit::BackendWithCompiler::execute(c10::IValue, c10::List<c10::IValue>): error: undefined reference to 'torch::jit::mobile::getCurrentEdgeProfiler()'
2021-10-07T22:54:20.9442293Z bazel-out/k8-fastbuild/bin/_objs/jit_tests/test_backend_compiler_lib.pic.o:test_backend_compiler_lib.cpp:function torch::jit::BackendWithCompiler::execute(c10::IValue, c10::List<c10::IValue>): error: undefined reference to 'torch::jit::mobile::getCurrentEdgeProfiler()'
2021-10-07T22:54:20.9446243Z bazel-out/k8-fastbuild/bin/_objs/jit_tests/test_backend_compiler_lib.pic.o:test_backend_compiler_lib.cpp:function torch::jit::BackendWithCompiler::execute(c10::IValue, c10::List<c10::IValue>): error: undefined reference to 'torch::jit::mobile::KinetoEdgeCPUProfiler::recordBackendEvent(long, long, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
2021-10-07T22:54:20.9448601Z collect2: error: ld returned 1 exit status
2021-10-07T22:54:20.9449473Z �[32m[2,855 / 2,863]�[0m 4 / 25 tests;�[0m 7 actions running�[0m; last test: �[32m//:memory_test�[0m
2021-10-07T22:54:20.9450300Z     Linking dataloader_test; 4s processwrapper-sandbox
2021-10-07T22:54:20.9451320Z     Testing //:enum_test; 2s processwrapper-sandbox
2021-10-07T22:54:20.9452425Z     Testing //:ordered_dict_test; 2s processwrapper-sandbox
2021-10-07T22:54:20.9453325Z     Testing //:misc_test; 2s processwrapper-sandbox
2021-10-07T22:54:20.9454042Z     Testing //:integration_test; 2s processwrapper-sandbox
2021-10-07T22:54:20.9454842Z     Compiling test/cpp/api/tensor.cpp; 2s processwrapper-sandbox
2021-10-07T22:54:20.9456113Z     Testing //:modules_test; 2s processwrapper-sandbox
2021-10-07T22:54:21.1899018Z 
2021-10-07T22:54:21.1900365Z �[1A�[K

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

kimishpatel · 2021-09-01T21:39:42Z

@kimishpatel has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

…al source" Summary: This diff exposes a way to add events to kineto profiler from external source. This can be a backend that executes a subgraph and wants to record this execution in kineto profiler. This diff also adds "backend" metadata to identify the backend an event would have executed on. Test Plan: test_lite_interpreter Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D30710710](https://our.internmc.facebook.com/intern/diff/D30710710) Here is a chrome trace output for example delegated backend ![image](https://user-images.githubusercontent.com/17488981/131749023-71062b4c-9d1f-428a-9bb1-6bd45b02d194.png) [ghstack-poisoned]

Summary: This diff exposes a way to add events to kineto profiler from external source. This can be a backend that executes a subgraph and wants to record this execution in kineto profiler. This diff also adds "backend" metadata to identify the backend an event would have executed on. Test Plan: test_lite_interpreter Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 11526ec Pull Request resolved: #64397

kimishpatel · 2021-09-01T21:44:53Z

@kimishpatel has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

raziel

LGTM Thanks

Though somebody from Kineto should review the Kineto changes.

raziel · 2021-09-03T01:10:45Z

torch/csrc/autograd/profiler_kineto.cpp

+  ctx_ptr->startThreadId = at::RecordFunction::currentThreadId();
+  ctx_ptr->debug_handle = debug_handle;
+
+  /* no support for input shapes now?


why's this?

Mainly to avoid overhead of shape profiling.

gdankel · 2021-09-04T18:30:50Z

torch/csrc/autograd/profiler_kineto.cpp

    }
-    std::string evt_name(fn.name().str());
-    auto end_time = getTimeUs();
+    auto end_time = ctx->endUS;


Can we assume that this is always set?

gdankel · 2021-09-04T18:31:40Z

torch/csrc/autograd/profiler_kineto.cpp

          }

-          state_ptr->reportClientActivity(fn, kineto_ctx_ptr);
+          kineto_ctx_ptr->endUS = getTimeUs();


Why move this here?

This is so that reportClientActivity can assume the start and end time are already encoded in ctx. When backend's report the runtime per op they will use indirectly use reportClientActivity. At the time of recording runtime of op, backends already have the duration/start and end time.

gdankel · 2021-09-04T18:40:59Z

torch/csrc/autograd/profiler_kineto.h

 #endif // USE_KINETO
 };

+TORCH_API void reportBackendEventToActiveKinetoProfiler(


Please add comment / documentation.

gdankel · 2021-09-09T12:17:34Z

torch/csrc/autograd/profiler_kineto.cpp

+  if (!state_ptr) {
+    return;
+  }
+  auto ctx_ptr = std::make_unique<KinetoObserverContext>();


While this is fine for the moment, I think we can optimize this a lot - it seems unnecessary to create the KinetoObserverContext in this case. In fact we can probably remove it from the profiler altogether...

I dont know if you meant for the backend event reporting only or generally in the profiler itself but I completely agree. I thought the same that we dont need the ObserverContext. My understanding was that the observer context was used by callback where enter callback construct one, push some info and return the context. Then record_function calls exit callback with the context which populates more information. For the purposes of the API here, it is not needed.

gdankel

I have some nits but in general this looks fine for now.
Definitely need to have a follow-up refactoring but that applies to the entire profiler.

…al source" Summary: This diff exposes a way to add events to kineto profiler from external source. This can be a backend that executes a subgraph and wants to record this execution in kineto profiler. This diff also adds "backend" metadata to identify the backend an event would have executed on. Test Plan: test_lite_interpreter Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D30710710](https://our.internmc.facebook.com/intern/diff/D30710710) Here is a chrome trace output for example delegated backend ![image](https://user-images.githubusercontent.com/17488981/131749023-71062b4c-9d1f-428a-9bb1-6bd45b02d194.png) [ghstack-poisoned]

pytorch-probot · 2021-10-05T21:50:26Z

CI Flow Status

⚛️ CI Flow

Ruleset - Version: v1
Ruleset - File: https://github.com/pytorch/pytorch/blob/3a28ee8661a33b29f0f2a1fad65d90b0c9529f00/.github/generated-ciflow-ruleset.json
PR ciflow labels: ciflow/default

Workflows	Labels (bold enabled)	Status
Triggered Workflows
linux-bionic-py3.6-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/noarch`, `ciflow/xla`	✅ triggered
linux-vulkan-bionic-py3.6-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/vulkan`	✅ triggered
linux-xenial-cuda11.3-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/linux`	✅ triggered
linux-xenial-py3.6-clang7-asan	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/sanitizers`	✅ triggered
linux-xenial-py3.6-clang7-onnx	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/onnx`	✅ triggered
linux-xenial-py3.6-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
linux-xenial-py3.6-gcc7-bazel-test	`ciflow/all`, `ciflow/bazel`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
periodic-pytorch-linux-xenial-cuda10.2-cudnn7-py3-gcc7-slow-gradcheck	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/linux`, `ciflow/slow`, `ciflow/slow-gradcheck`	✅ triggered
win-vs2019-cpu-py3	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/win`	✅ triggered
win-vs2019-cuda11.3-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/win`	✅ triggered
Skipped Workflows
libtorch-linux-xenial-cuda10.2-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`	🚫 skipped
libtorch-linux-xenial-cuda11.3-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`	🚫 skipped
linux-bionic-cuda10.2-py3.9-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/slow`	🚫 skipped
linux-xenial-cuda10.2-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/slow`	🚫 skipped
parallelnative-linux-xenial-py3.6-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`	🚫 skipped
periodic-libtorch-linux-xenial-cuda11.1-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-linux-xenial-cuda11.1-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-win-vs2019-cuda11.1-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/scheduled`, `ciflow/win`	🚫 skipped
puretorch-linux-xenial-py3.6-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`	🚫 skipped

You can add a comment to the PR and tag @pytorchbot with the following commands:

# ciflow rerun, "ciflow/default" will always be added automatically
@pytorchbot ciflow rerun

# ciflow rerun with additional labels "-l <ciflow/label_name>", which is equivalent to adding these labels manually and trigger the rerun
@pytorchbot ciflow rerun -l ciflow/scheduled -l ciflow/slow

For more information, please take a look at the CI Flow Wiki.

Summary: This diff exposes a way to add events to kineto profiler from external source. This can be a backend that executes a subgraph and wants to record this execution in kineto profiler. This diff also adds "backend" metadata to identify the backend an event would have executed on. Test Plan: test_lite_interpreter Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: c6e9359 Pull Request resolved: #64397

kimishpatel · 2021-10-05T22:16:33Z

@kimishpatel has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

…al source" Summary: This diff exposes a way to add events to kineto profiler from external source. This can be a backend that executes a subgraph and wants to record this execution in kineto profiler. This diff also adds "backend" metadata to identify the backend an event would have executed on. Test Plan: test_lite_interpreter Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D30710710](https://our.internmc.facebook.com/intern/diff/D30710710) Here is a chrome trace output for example delegated backend ![image](https://user-images.githubusercontent.com/17488981/131749023-71062b4c-9d1f-428a-9bb1-6bd45b02d194.png) [ghstack-poisoned]

Summary: This diff exposes a way to add events to kineto profiler from external source. This can be a backend that executes a subgraph and wants to record this execution in kineto profiler. This diff also adds "backend" metadata to identify the backend an event would have executed on. Test Plan: test_lite_interpreter Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 3942222 Pull Request resolved: #64397

janeyx99 · 2021-10-09T00:45:55Z

Reverting this PR as it failed the bazel test (PR signal was red as well) on HUD

facebook-github-bot · 2021-10-09T00:47:52Z

This pull request has been reverted by c62ed96. To re-land this change, follow these steps.

kimishpatel · 2021-10-09T01:04:09Z

Reverting this PR as it failed the bazel test (PR signal was red as well) on HUD

Is the failing build trigerred on the PRs? I remember seeing all green for this PR's CI.

janeyx99 · 2021-10-12T19:31:12Z

Yes, it is. On your diff and in the HUD PR page, I see:

kimishpatel requested review from albanD and soulitzer as code owners September 1, 2021 21:24

facebook-github-bot added cla signed oncall: jit Add this issue/PR to JIT oncall triage queue labels Sep 1, 2021

albanD removed their request for review September 1, 2021 21:28

kimishpatel requested review from gdankel, ilia-cher and raziel September 1, 2021 22:07

raziel approved these changes Sep 3, 2021

View reviewed changes

gdankel reviewed Sep 4, 2021

View reviewed changes

gdankel reviewed Sep 9, 2021

View reviewed changes

gdankel approved these changes Sep 9, 2021

View reviewed changes

pytorch-probot bot added the ciflow/default label Oct 5, 2021

facebook-github-bot closed this in c1343ff Oct 8, 2021

facebook-github-bot added the Reverted label Oct 9, 2021

janeyx99 mentioned this pull request Oct 9, 2021

[Meta] CI Revert Tracker #66178

Closed

facebook-github-bot deleted the gh/kimishpatel/78/head branch October 12, 2021 14:18

[Pytorch Edge] Support profiling kineto events from external source #64397

[Pytorch Edge] Support profiling kineto events from external source #64397

Uh oh!

Conversation

kimishpatel commented Sep 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Sep 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

🕵️ 1 new failure recognized by patterns

linux-xenial-py3.6-gcc7-bazel-test / build-and-test (1/1)

Uh oh!

kimishpatel commented Sep 1, 2021

Uh oh!

kimishpatel commented Sep 1, 2021

Uh oh!

raziel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gdankel left a comment

Choose a reason for hiding this comment

Uh oh!

pytorch-probot bot commented Oct 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚛️ CI Flow

Uh oh!

kimishpatel commented Oct 5, 2021

Uh oh!

janeyx99 commented Oct 9, 2021

Uh oh!

facebook-github-bot commented Oct 9, 2021

Uh oh!

kimishpatel commented Oct 9, 2021

Uh oh!

janeyx99 commented Oct 12, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kimishpatel commented Sep 1, 2021 •

edited

Loading

facebook-github-bot commented Sep 1, 2021 •

edited

Loading

pytorch-probot bot commented Oct 5, 2021 •

edited

Loading