Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refine the spill threshold for aggregation/sort, refine the memory usage calculation for aggregation #6708

Merged
merged 11 commits into from
Feb 9, 2023
Merged

Refine the spill threshold for aggregation/sort, refine the memory usage calculation for aggregation #6708

merged 11 commits into from
Feb 9, 2023

Conversation

windtalker
Copy link
Contributor

What problem does this PR solve?

Issue Number: ref #6528

Problem Summary:

What is changed and how it works?

  1. refine the spill threshold for aggregation/sort, now if aggregation/sort is executed parallelly, the threshold used in every sub agg/sort is recalculated by total threshold and the concurrency
  2. refine the memory usage calculation for aggregation, don't rely on the task-level memory tracker to calculate the memory usage of aggregation.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Feb 1, 2023 •
edited

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • SeaRise
  • yibin87

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 1, 2023
@SeaRise SeaRise self-requested a review February 1, 2023 08:02
@yibin87 yibin87 self-requested a review February 1, 2023 08:04
yibin87
yibin87 reviewed Feb 1, 2023
View reviewed changes
dbms/src/Interpreters/Aggregator.h
@@ -718,7 +718,7 @@ struct AggregatedDataVariants : private boost::noncopyable
case Type::NAME: \
{ \
const auto * ptr = reinterpret_cast<const AggregationMethodName(NAME) *>(aggregation_method_impl); \
return ptr->data.size() + (without_key != nullptr); \
return ptr->data.size(); \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why remove the (without_key != nullptr) check here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is used by over_flow_row, and we already remove the support of over_flow_row

yibin87
yibin87 approved these changes Feb 1, 2023
View reviewed changes
Copy link
Contributor

@yibin87 yibin87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Feb 1, 2023
SeaRise
SeaRise reviewed Feb 1, 2023
View reviewed changes
dbms/src/Flash/Coprocessor/InterpreterUtils.cpp Outdated
@@ -120,7 +120,7 @@ void orderStreams(
order_descr,
settings.max_block_size,
limit,
settings.max_bytes_before_external_sort,
getAverageThreshold(settings.max_bytes_before_external_sort, std::min(max_streams, pipeline.streams.size())),
Copy link
Contributor

@SeaRise SeaRise Feb 1, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer

auto avg_threshold = getAverageThreshold(settings.max_bytes_before_external_sort, pipeline.streams.size());
pipeline.transform([&](auto & stream) {
    stream = std::make_shared<MergeSortingBlockInputStream>(xxxx, avg_threshold);
};

The count of sort stream is pipeline.streams.size() at this time, not min(max_streams, pipeline.streams.size()), even if max_streams < pipeline.streams.size(). So we shoud use pipeline.streams.size() or not std::min(max_streams, pipeline.streams.size()) here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@SeaRise SeaRise self-requested a review February 6, 2023 07:00
SeaRise
SeaRise approved these changes Feb 6, 2023
View reviewed changes
Copy link
Contributor

@SeaRise SeaRise left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Feb 6, 2023
@windtalker
Copy link
Contributor Author

/run-all-tests

@windtalker
Copy link
Contributor Author

/hold

@ti-chi-bot ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 7, 2023
@windtalker
Copy link
Contributor Author

windtalker commented Feb 7, 2023 •
edited

hold after code freeze for v6.6

@windtalker
AggregatedDataVariants maintain its own memory usage
6b0cc78 
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
@windtalker
more
28e48f5 
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
@windtalker
fix
36b0b76 
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
@windtalker
refine
689e589 
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
@windtalker
fix bug
fda2274 
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
@windtalker
address comments
80f8537 
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
@windtalker
fix
d12f1c0 
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
@windtalker
fix
e62a51d 
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
@windtalker
refine
924b668 
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
@windtalker
fix build
8fbf2c2 
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
@windtalker
Copy link
Contributor Author

/unhold

@ti-chi-bot ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 9, 2023
@windtalker
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

@windtalker: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 8fbf2c2

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Feb 9, 2023
@ti-chi-bot
Copy link
Member

@windtalker: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

trigger some heavy tests which will not run always when PR updated.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot merged commit 8942a86 into pingcap:master Feb 9, 2023
guo-shaoge pushed a commit to guo-shaoge/tiflash that referenced this pull request Feb 10, 2023
@windtalker @guo-shaoge
Refine the spill threshold for aggregation/sort, refine the memory us…
ff2949f 
…age calculation for aggregation (pingcap#6708)

ref pingcap#6528
@windtalker windtalker deleted the agg_memory_usage branch May 6, 2023 06:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yibin87 yibin87 yibin87 approved these changes

@SeaRise SeaRise SeaRise approved these changes

Assignees
No one assigned
Labels
release-note-none size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@windtalker @ti-chi-bot @yibin87 @SeaRise