Add the parsing part for the profiling tool #7043

kuke · 2017-12-26T12:36:55Z

See Part I in #6701

Please review this part after the Part I merged.

… profiler_tool

kuke · 2018-01-05T08:57:01Z

The profiling report looks like:

W0105 14:48:05.080145  1552 profiler.cc:232] Cannot find the push marker of event 'event_without_push', which will be ignored in profiling report.
W0105 14:48:05.080199  1552 profiler.cc:266] Cannot find the pop marker of event 'event_without_pop', which will be ignored in profiling report.

------------------------->     Profiling Report     <-------------------------

Place: CUDA
Time unit: ms
Sorted by total time in descending order in the same thread

Event                Calls       Total       Min.        Max.        Ave.
thread0::op_1        3           0.006565    0.001816    0.002444    0.00218833
thread0::op_3        3           0.005658    0.001816    0.001956    0.001886
thread0::op_4        3           0.005518    0.001816    0.001886    0.00183933
thread0::op_2        3           0.005446    0.001815    0.001816    0.00181533
thread0::evs_op_3    1           0.002165    0.002165    0.002165    0.002165
thread0::evs_op_2    1           0.002026    0.002026    0.002026    0.002026
thread0::evs_op_1    1           0.001956    0.001956    0.001956    0.001956
thread0::evs_op_4    1           0.001816    0.001816    0.001816    0.001816

And this pull request is ready for review. @qingqing01 @reyoung

qingqing01 · 2018-01-05T10:04:21Z

paddle/platform/profiler.cc

+void PopEvent(const std::string& name, DeviceContext* dev_ctx) {
+  GetEventList().Record(EventKind::kPopRange, std::move(name), g_thread_id,
+                        dev_ctx);
+}


Need to refine RecordEvent::RecordEvent() and RecordEvent::~RecordEvent() to reuse these two functions.

qingqing01 · 2018-01-05T10:08:09Z

paddle/platform/profiler.cc

+  std::cout << "Place: GPU" << std::endl;
+#else
+  std::cout << "Place: CPU" << std::endl;
+#endif


If compiling with GPU, the users also can run tasks on CPU only. So here should not use the PADDLE_WITH_CUDA macro. You can use g_state to determine.

Done. Use another global variable g_profiler_place because g_state has been set to kDisabled when parsing events.

qingqing01 · 2018-01-05T10:10:14Z

paddle/platform/profiler.cc

+          double event_time = rit->CudaElapsedMs(events[i][j]);
+#else
+          double event_time = rit->CpuElapsedMs(events[i][j]);
+#endif


The same problem with above.

qingqing01 · 2018-01-05T10:12:13Z

paddle/platform/profiler.cc

+          pushed_events.erase((++rit).base());
+        } else {
+          std::cout << "Warning: can not find the start marker of event "
+                    << events[i][j].name();


LOG(WARNING)

qingqing01 · 2018-01-05T10:13:31Z

paddle/platform/profiler.cc

+                << std::setw(data_width) << event_item.max_time
+                << std::setw(data_width) << event_item.ave_time << std::endl;
+    }
+  }


Maybe the ParseEvents and print can be two functions.

kuke

Followed all the comments. Thanks!

kuke · 2018-01-05T14:54:44Z

paddle/platform/profiler.cc

+void PopEvent(const std::string& name, DeviceContext* dev_ctx) {
+  GetEventList().Record(EventKind::kPopRange, std::move(name), g_thread_id,
+                        dev_ctx);
+}


kuke · 2018-01-05T14:54:52Z

paddle/platform/profiler.cc

+  std::cout << "Place: GPU" << std::endl;
+#else
+  std::cout << "Place: CPU" << std::endl;
+#endif


Done. Use another global variable g_profiler_place because g_state has been set to kDisabled when parsing events.

kuke · 2018-01-05T14:55:00Z

paddle/platform/profiler.cc

+          double event_time = rit->CudaElapsedMs(events[i][j]);
+#else
+          double event_time = rit->CpuElapsedMs(events[i][j]);
+#endif


kuke · 2018-01-05T14:55:06Z

paddle/platform/profiler.cc

+          pushed_events.erase((++rit).base());
+        } else {
+          std::cout << "Warning: can not find the start marker of event "
+                    << events[i][j].name();


kuke · 2018-01-05T14:55:16Z

paddle/platform/profiler.cc

+                << std::setw(data_width) << event_item.max_time
+                << std::setw(data_width) << event_item.ave_time << std::endl;
+    }
+  }


qingqing01 · 2018-01-08T03:13:48Z

paddle/platform/profiler.cc

@@ -13,12 +13,18 @@ See the License for the specific language governing permissions and
 limitations under the License. */

 #include "paddle/platform/profiler.h"
+#include <iomanip>
+#include <map>
+#include "gflags/gflags.h"


The gflags is not used. This header can be removed.

qingqing01 · 2018-01-08T03:37:19Z

paddle/platform/profiler.cc

+                      return a.max_time > b.max_time;
+                    default:
+                      return a.ave_time > b.ave_time;
+                  }


Maybe this lambda expression can be selected before the for loop in line 187, and the sort_domain can be saved at the same time. Then pass the sort_domain as a string argument to PrintProfilingReport function.

kuke

Modified. Thanks!

kuke · 2018-01-08T04:36:38Z

paddle/platform/profiler.cc

@@ -13,12 +13,18 @@ See the License for the specific language governing permissions and
 limitations under the License. */

 #include "paddle/platform/profiler.h"
+#include <iomanip>
+#include <map>
+#include "gflags/gflags.h"


kuke · 2018-01-08T04:36:49Z

paddle/platform/profiler.cc

+                      return a.max_time > b.max_time;
+                    default:
+                      return a.ave_time > b.ave_time;
+                  }


qingqing01

LGTM.

chengduoZH · 2018-01-09T12:32:19Z

paddle/platform/profiler.cc

 #ifdef PADDLE_WITH_CUDA
  PADDLE_ENFORCE(e.has_cuda() && has_cuda());
  PADDLE_ENFORCE(e.device() == device());
  PADDLE_ENFORCE(cudaEventSynchronize(event_));
  PADDLE_ENFORCE(cudaEventSynchronize(e.event()));
  float ms;
  PADDLE_ENFORCE(cudaEventElapsedTime(&ms, event_, e.event()));
-  return ms * 1000.0;
+  return ms;
 #else
  PADDLE_THROW("CUDA is not enabled");
 #endif


I can not comment on line 110 but only write the comments here.
line 110 should be before line 109.

chengduoZH · 2018-01-09T13:06:55Z

paddle/platform/profiler.cc


 namespace paddle {
 namespace platform {

 // The profiler state, the initial value is ProfilerState::kDisabled
 static ProfilerState g_state = ProfilerState::kDisabled;
+// To record which timer the profiler used, CUDA or CPU.
+static std::string g_profiler_place = "";


g_profiler_place and g_state should be thread_local, just like g_thread_id.

chengduoZH · 2018-01-09T13:08:55Z

paddle/platform/profiler.cc

-      name_(std::move(name)),
-      thread_id_(thread_id),
-      has_cuda_(false) {
+    : kind_(kind), name_(name), thread_id_(thread_id), has_cuda_(false) {


Why do not you use std::move?

chengduoZH · 2018-01-09T13:12:52Z

paddle/platform/profiler.cc

-double Event::CpuElapsedUs(const Event& e) const {
-  return (e.cpu_ns_ - cpu_ns_) / (1000.0);
+double Event::CpuElapsedMs(const Event& e) const {
+  return (e.cpu_ns_ - cpu_ns_) / (1000000.0);


Ii will be better to replace / (1000000.0) with *0.000001.

Yibing Liu added 2 commits December 26, 2017 12:28

Add the parsing part for the profiling tool

55b17c1

Merge branch 'develop' of upstream into profiler_tool

5837166

kuke requested review from reyoung and qingqing01 December 26, 2017 12:39

Yibing Liu added 5 commits December 27, 2017 02:54

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

24458ae

… profiler_tool

Merge the update in profiling tool

770aff2

Confirm the contents in profiling report

0f44107

Format profiling report

2d94eca

Enable sorting the profiling result by different keys

0aa03a8

Yibing Liu added 3 commits January 5, 2018 09:04

fix typos

d7e5684

Make time calc funcs return ms instead of us

5a0a461

Merge branch 'develop' of upstream into profiler_tool

367a5c9

qingqing01 reviewed Jan 5, 2018

View reviewed changes

Fix bad_alloc bug & refine code in profiler

df3b250

kuke commented Jan 5, 2018

View reviewed changes

Fix profiler place bug

7a4f3be

qingqing01 reviewed Jan 8, 2018

View reviewed changes

Yibing Liu added 3 commits January 8, 2018 04:29

Remove the redundant switch case statement

d09503b

Remove unused included header gflags

ea0280b

Merge branch 'develop' of upstream into profiling_tool

6416eee

kuke commented Jan 8, 2018

View reviewed changes

qingqing01 approved these changes Jan 8, 2018

View reviewed changes

kuke merged commit 5e90f5e into PaddlePaddle:develop Jan 8, 2018

chengduoZH reviewed Jan 9, 2018

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the parsing part for the profiling tool #7043

Add the parsing part for the profiling tool #7043

kuke commented Dec 26, 2017 •

edited

kuke commented Jan 5, 2018 •

edited

qingqing01 Jan 5, 2018

kuke Jan 5, 2018

qingqing01 Jan 5, 2018

kuke Jan 5, 2018

qingqing01 Jan 5, 2018

kuke Jan 5, 2018

qingqing01 Jan 5, 2018

kuke Jan 5, 2018

qingqing01 Jan 5, 2018

kuke Jan 5, 2018

kuke left a comment

kuke Jan 5, 2018

kuke Jan 5, 2018

kuke Jan 5, 2018

kuke Jan 5, 2018

kuke Jan 5, 2018

qingqing01 Jan 8, 2018

kuke Jan 8, 2018

qingqing01 Jan 8, 2018

kuke Jan 8, 2018

kuke left a comment

kuke Jan 8, 2018

kuke Jan 8, 2018

qingqing01 left a comment

chengduoZH Jan 9, 2018

chengduoZH Jan 9, 2018

chengduoZH Jan 9, 2018

chengduoZH Jan 9, 2018

Add the parsing part for the profiling tool #7043

Add the parsing part for the profiling tool #7043

Conversation

kuke commented Dec 26, 2017 • edited

kuke commented Jan 5, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kuke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kuke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qingqing01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kuke commented Dec 26, 2017 •

edited

kuke commented Jan 5, 2018 •

edited