Fixing duplication in operator profiling #15240

Zha0q1 · 2019-06-13T20:13:10Z

Description

fix: #10520
fix: #15243
For detailed explanations of the cause as well the proposed fix of the issue, please refer to https://cwiki.apache.org/confluence/display/MXNET/Fixing+Duplication+in+Operator+Profiling?moved=true

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

vandanavk · 2019-06-14T21:48:22Z

@mxnet-label-bot add [Profiler, pr-awaiting-review]

sandeep-krishnamurthy

Can we add few test case?

src/profiler/profiler.h

Zha0q1 · 2019-06-14T23:36:31Z

Can we add few test case?

Yeah, after we merge #15132. This way, I can use the json return to test.

access2rohit · 2019-06-20T18:14:40Z

tests/python/unittest/test_profiler.py

+        and 'Count' in target_dict['Time']['operator']['sqrt'] \
+        and '_plus_scalar' in target_dict['Time']['operator'] \
+        and 'Count' in target_dict['Time']['operator']['_plus_scalar']
+    # thet are called once and twice respectively


typo: they ?

Fixed. Nice catch!

access2rohit · 2019-06-20T18:16:35Z

LGTM! just fix the typo in comments

sandeep-krishnamurthy

One minor comment.

sandeep-krishnamurthy · 2019-06-20T19:40:04Z

src/profiler/profiler.h

@@ -843,6 +854,8 @@ struct ProfileTask : public ProfileDuration {
  VTUNE_ONLY_CODE(std::unique_ptr<vtune::VTuneTask> vtune_task_);
  /*! \brief NVTX duration object */
  NVTX_ONLY_CODE(std::unique_ptr<nvtx::NVTXDuration> nvtx_duration_);
+  /*! \brief not to add this stat to AggregateStats */


nit: 'not to add...' -> Add profiler stat to AggregateStats.

sandeep-krishnamurthy · 2019-06-20T19:42:32Z

src/profiler/profiler.h

+  /*!
+   * \brief Whether to add stat to AggregateStats
+   */
+  void enableAggregateStats(bool enabled) {


In the all the places this is True. Except in VTune task case.
Can we make it default parameter with default value True?
So in only VTune Task case, we set it to False. In other places, we don't need to make changes.

I can default enabled to true. enableAggregateStats() is a new function I added and the only place it is called is in line 1167 with enabled == false

enable_aggregate_ has already been defaulted to true

sandeep-krishnamurthy

Thanks. LGTM

sandeep-krishnamurthy · 2019-06-21T16:24:36Z

Hi @Zha0q1, I assume you are closing and opening PR to pass CI?
Are tests passing locally on your machine? If tests are flaky is there a Github issue to track it?
Because, each CI run is very costly.

Zha0q1 · 2019-06-21T17:23:27Z

Hi @Zha0q1, I assume you are closing and opening PR to pass CI?
Are tests passing locally on your machine? If tests are flaky is there a Github issue to track it?
Because, each CI run is very costly.

Sorry I will use CI more frugally. Yes they are passing locally.
Yesterday some of the tests were not assigned a worker thread and the time limit was exceeded, so I restarted the tests

sandeep-krishnamurthy · 2019-06-21T20:17:30Z

Hi @Zha0q1, I assume you are closing and opening PR to pass CI?
Are tests passing locally on your machine? If tests are flaky is there a Github issue to track it?
Because, each CI run is very costly.

Sorry I will use CI more frugally. Yes they are passing locally.
Yesterday some of the tests were not assigned a worker thread and the time limit was exceeded, so I restarted the tests

Thanks. No problem. If it is passing locally and failing in CI, then CI should help us. Please file Github issue if you see Flaky issues so we can fix it.

Zha0q1 changed the title ~~Fixing duplication in operator profiling~~ [WIP] Fixing duplication in operator profiling Jun 13, 2019

Zha0q1 mentioned this pull request Jun 13, 2019

Profiler: Operator Aggregate Stats Duplication #15243

Closed

Zha0q1 changed the title ~~[WIP] Fixing duplication in operator profiling~~ Fixing duplication in operator profiling Jun 14, 2019

marcoabreu added pr-awaiting-review PR is waiting for code review Profiler MXNet profiling issues labels Jun 14, 2019

sandeep-krishnamurthy reviewed Jun 14, 2019

View reviewed changes

src/profiler/profiler.h Outdated Show resolved Hide resolved

src/profiler/profiler.h Outdated Show resolved Hide resolved

Zha0q1 added 6 commits June 17, 2019 13:48

initial

d3534bd

adding a bool in ProfileStat to control if add to aggregate stats

1cd9fbd

add test case

65469b0

fix style

48e888f

stylefix

5b23b47

testcases

60607e5

access2rohit reviewed Jun 20, 2019

View reviewed changes

access2rohit approved these changes Jun 20, 2019

View reviewed changes

fix type

34c6cc1

sandeep-krishnamurthy reviewed Jun 20, 2019

View reviewed changes

fix comment

5980b95

sandeep-krishnamurthy approved these changes Jun 20, 2019

View reviewed changes

sandeep-krishnamurthy added pr-awaiting-merge Review and CI is complete. Ready to Merge and removed pr-awaiting-review PR is waiting for code review labels Jun 20, 2019

Zha0q1 closed this Jun 20, 2019

Zha0q1 reopened this Jun 20, 2019

Zha0q1 closed this Jun 20, 2019

Zha0q1 reopened this Jun 20, 2019

Zha0q1 closed this Jun 21, 2019

Zha0q1 reopened this Jun 21, 2019

Update profiler.h

1167c8b

Zha0q1 closed this Jun 21, 2019

Zha0q1 reopened this Jun 21, 2019

sandeep-krishnamurthy merged commit 3f8fd00 into apache:master Jun 21, 2019

roywei mentioned this pull request Jun 25, 2019

[CI] nightly failure test on amp tutorial #15355

Closed

roywei mentioned this pull request Jul 5, 2019

Operator Performance Regression on CPU #15429

Open

6 tasks

eric-haibin-lin mentioned this pull request Sep 20, 2019

[Discussion] 1.5.1 Patch Release #15613

Closed

leezu mentioned this pull request Jul 20, 2020

pytest worker crash seen on newly introduced unittest test_profiler_gpu.py::test_aggregate_duplication #18756

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixing duplication in operator profiling #15240

Fixing duplication in operator profiling #15240

Zha0q1 commented Jun 13, 2019 •

edited

vandanavk commented Jun 14, 2019

sandeep-krishnamurthy left a comment

Zha0q1 commented Jun 14, 2019

access2rohit Jun 20, 2019

Zha0q1 Jun 20, 2019

access2rohit commented Jun 20, 2019

sandeep-krishnamurthy left a comment

sandeep-krishnamurthy Jun 20, 2019

Zha0q1 Jun 20, 2019

sandeep-krishnamurthy Jun 20, 2019

Zha0q1 Jun 20, 2019 •

edited

Zha0q1 Jun 20, 2019

sandeep-krishnamurthy left a comment

sandeep-krishnamurthy commented Jun 21, 2019

Zha0q1 commented Jun 21, 2019 •

edited

sandeep-krishnamurthy commented Jun 21, 2019

Fixing duplication in operator profiling #15240

Fixing duplication in operator profiling #15240

Conversation

Zha0q1 commented Jun 13, 2019 • edited

Description

Checklist

Essentials

vandanavk commented Jun 14, 2019

sandeep-krishnamurthy left a comment

Choose a reason for hiding this comment

Zha0q1 commented Jun 14, 2019

access2rohit Jun 20, 2019

Choose a reason for hiding this comment

Zha0q1 Jun 20, 2019

Choose a reason for hiding this comment

access2rohit commented Jun 20, 2019

sandeep-krishnamurthy left a comment

Choose a reason for hiding this comment

sandeep-krishnamurthy Jun 20, 2019

Choose a reason for hiding this comment

Zha0q1 Jun 20, 2019

Choose a reason for hiding this comment

sandeep-krishnamurthy Jun 20, 2019

Choose a reason for hiding this comment

Zha0q1 Jun 20, 2019 • edited

Choose a reason for hiding this comment

Zha0q1 Jun 20, 2019

Choose a reason for hiding this comment

sandeep-krishnamurthy left a comment

Choose a reason for hiding this comment

sandeep-krishnamurthy commented Jun 21, 2019

Zha0q1 commented Jun 21, 2019 • edited

sandeep-krishnamurthy commented Jun 21, 2019

Zha0q1 commented Jun 13, 2019 •

edited

Zha0q1 Jun 20, 2019 •

edited

Zha0q1 commented Jun 21, 2019 •

edited