Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

executor: add memTracker for aggregation #13720

Merged
merged 4 commits into from Dec 25, 2019
Merged

executor: add memTracker for aggregation #13720

merged 4 commits into from Dec 25, 2019

Conversation

ichn-hu
Copy link
Contributor

@ichn-hu ichn-hu commented Nov 25, 2019

What problem does this PR solve?

Add memory track for aggregation, close #13392.

What is changed and how it works?

By tracking the chunks read from child, it is fair easy to track the memory usage (partially).

Check List

Tests

  • Unit test

Side effects

  • Possible performance regression

Release note

  • Write release note for bug-fix or new feature.

@ichn-hu ichn-hu changed the title Hash agg memtrack executor: add memtrack for aggregation Nov 25, 2019
@codecov
Copy link

codecov bot commented Nov 25, 2019 •
edited

Codecov Report

❗ No coverage uploaded for pull request base (master@f9cdebd). Click here to learn what that means.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #13720   +/-   ##
===========================================
  Coverage          ?   80.5015%           
===========================================
  Files             ?        483           
  Lines             ?     123615           
  Branches          ?          0           
===========================================
  Hits              ?      99512           
  Misses            ?      16203           
  Partials          ?       7900

@zyxbest
Copy link
Contributor

zyxbest commented Nov 26, 2019

/run-all-tests

1 similar comment
@ichn-hu
Copy link
Contributor Author

ichn-hu commented Nov 26, 2019

/run-all-tests

XuHuaiyu
XuHuaiyu reviewed Nov 27, 2019
View reviewed changes
executor/aggregate.go Outdated Show resolved Hide resolved
@qw4990 qw4990 self-requested a review December 4, 2019 12:59
qw4990
qw4990 reviewed Dec 4, 2019
View reviewed changes
Copy link
Contributor

@qw4990 qw4990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All intermediate data is stored in aggPartialResultMapper at https://github.com/pingcap/tidb/blob/master/executor/aggregate.go#L467 .
It seems we should trace it too.

@ichn-hu
Copy link
Contributor Author

ichn-hu commented Dec 6, 2019

All intermediate data is stored in aggPartialResultMapper at https://github.com/pingcap/tidb/blob/master/executor/aggregate.go#L467 .
It seems we should trace it too.

You are right, but the thing is we current lack the memory usage API for unsafe.Pointer, which is how aggfuncs.PartialResult is implemented, therefore it would be kind of difficult to do so. Perhaps we could do it in a future PR?

@ichn-hu
Copy link
Contributor Author

ichn-hu commented Dec 6, 2019

@XuHuaiyu @qw4990 PTAL again.

@ichn-hu ichn-hu requested review from a team as code owners December 6, 2019 07:16
@ghost ghost requested review from alivxxx, lzmhhh123 and qw4990 and removed request for a team December 6, 2019 07:16
@ichn-hu
Copy link
Contributor Author

ichn-hu commented Dec 6, 2019

Sorry, my bad...

@SunRunAway SunRunAway removed their request for review December 9, 2019 10:47
@sre-bot
Copy link
Contributor

sre-bot commented Dec 16, 2019

@ichn-hu, please update your pull request.

@ichn-hu
Copy link
Contributor Author

ichn-hu commented Dec 17, 2019

/run-all-tests

1 similar comment
@zyxbest
Copy link
Contributor

zyxbest commented Dec 17, 2019

/run-all-tests

@zyxbest
Copy link
Contributor

zyxbest commented Dec 17, 2019

/rebuild

@zyxbest
Copy link
Contributor

zyxbest commented Dec 17, 2019

/run-integration-tests

@ichn-hu ichn-hu mentioned this pull request Dec 18, 2019
Closed
@ichn-hu
Copy link
Contributor Author

ichn-hu commented Dec 18, 2019 •
edited

@SunRunAway @XuHuaiyu @qw4990 Hi all, I've created another issue that aims to track the memory consumpution in partialResultMapper #14103, in this PR let's just track the memory usage of chunks. PTAL.

qw4990
qw4990 reviewed Dec 19, 2019
View reviewed changes
executor/aggregate.go Outdated Show resolved Hide resolved
qw4990
qw4990 reviewed Dec 19, 2019
View reviewed changes
executor/aggregate.go Outdated Show resolved Hide resolved
executor/aggregate.go Outdated Show resolved Hide resolved
qw4990
qw4990 reviewed Dec 23, 2019
View reviewed changes
executor/benchmark_test.go Outdated Show resolved Hide resolved
@qw4990
Copy link
Contributor

qw4990 commented Dec 23, 2019

@ichn-hu Friendly ping, please resolve conflicts.

@ichn-hu
Copy link
Contributor Author

ichn-hu commented Dec 23, 2019

/run-all-tests

@ichn-hu
Copy link
Contributor Author

ichn-hu commented Dec 23, 2019

@qw4990 @XuHuaiyu PTAL, this PR has been around for several weeks, let's merge it soon.

qw4990
qw4990 reviewed Dec 23, 2019
View reviewed changes
Copy link
Contributor

@qw4990 qw4990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@qw4990 qw4990 added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 23, 2019
XuHuaiyu
XuHuaiyu approved these changes Dec 25, 2019
View reviewed changes
Copy link
Contributor

@XuHuaiyu XuHuaiyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@XuHuaiyu XuHuaiyu changed the title executor: add memtrack for aggregation executor: add memTracker for aggregation Dec 25, 2019
@XuHuaiyu XuHuaiyu added status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 25, 2019
@sre-bot
Copy link
Contributor

sre-bot commented Dec 25, 2019

/run-all-tests

@sre-bot sre-bot merged commit efd34cb into pingcap:master Dec 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qw4990 qw4990 qw4990 left review comments

@XuHuaiyu XuHuaiyu XuHuaiyu approved these changes

Assignees
No one assigned
Labels
sig/execution SIG execution status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

memory tracking for aggregation executor.
6 participants
@ichn-hu @zyxbest @sre-bot @qw4990 @XuHuaiyu @SunRunAway