Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](compaction) support vertical_compaction & ordered_data_compaction #14524

Merged
merged 20 commits into from
Dec 1, 2022

Conversation

yixiutt
Copy link
Contributor

@yixiutt yixiutt commented Nov 23, 2022

  1. support vertical compaction, memory usage is 1/10 of horizontal compaction in ckbench.

2.ordered compaction, save min max key for segment in rowset meta, link file in compaction if rowsets are none overlapping.

Issue Number: close #xxx

Problem summary

Describe your changes.

Checklist(Required)

  1. Does it affect the original behavior:
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:
    • Yes
    • No
    • No Need
  3. Has document been added or modified:
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies:
    • Yes
    • No
  5. Are there any changes that cannot be rolled back:
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

yixiutt and others added 17 commits November 23, 2022 14:25
…e#13495)

1.remove quick_compaction's rowset pick policy, call cu compaction when trigger
quick compaction
2. skip tablet's compaction task when compaction score is too small

Co-authored-by: yixiutt <yixiu@selectdb.com>
@github-actions github-actions bot added area/vectorization kind/docs Categorizes issue or PR as related to documentation. kind/test labels Nov 23, 2022
@yixiutt yixiutt changed the title Compaction opt 1 [feature](compaction) some compaction feature Nov 23, 2022
@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 23, 2022

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.18 seconds
load time: 439 seconds
storage size: 17123485660 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221130134042_clickbench_pr_55623.html

@yixiutt yixiutt changed the title [feature](compaction) some compaction feature [feature](compaction) support vertical_compaction & ordered_data_compaction Nov 24, 2022
@yixiutt yixiutt force-pushed the compaction_opt_1 branch 2 times, most recently from 501559b to f12d95c Compare November 30, 2022 11:40
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
Copy link
Contributor

Passes P2, so we can merge it quickly.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 1, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Dec 1, 2022

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 1, 2022

PR approved by anyone and no changes requested.

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 94a6ffb into apache:master Dec 1, 2022
@zhangm365
Copy link

zhangm365 commented May 14, 2024

Is the variable enable_vertical_compaction set to true by default? That is set to enable the algorithm config.
Thanks a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. area/vectorization kind/docs Categorizes issue or PR as related to documentation. kind/test reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants