Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix](clean trash) Fix clean trash use agent task #33912

Merged
merged 5 commits into from
Apr 22, 2024

Conversation

deardeng
Copy link
Contributor

Proposed changes

clean trash 使用be的 rpc 线程干活,影响到了create tablet agent task执行,导致建表超时

本地复现:

在be代码void BackendService::clean_trash() 函数中sleep 100s
create database db1;
建表 如回归case中的
DROP database if EXISTS db1 force;
ADMIN CLEAN TRASH
create database db2; use db2;
CREATE TABLE IF NOT EXISTS `customer` (
  `c_custkey` int(11) NOT NULL COMMENT "",
  `c_name` varchar(26) NOT NULL COMMENT "",
  `c_address` varchar(41) NOT NULL COMMENT "",
  `c_city` varchar(11) NOT NULL COMMENT "",
  `c_nation` varchar(16) NOT NULL COMMENT "",
  `c_region` varchar(13) NOT NULL COMMENT "",
  `c_phone` varchar(16) NOT NULL COMMENT "",
  `c_mktsegment` varchar(11) NOT NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(`c_custkey`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 12
PROPERTIES (
"replication_num" = "1",
"colocate_with" = "groupa2",
"in_memory" = "false",
"storage_format" = "DEFAULT"
)

clean trash 改成使用agent task形式,下发任务

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@deardeng
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.42% (8922/25190)
Line Coverage: 27.12% (73290/270218)
Region Coverage: 26.26% (37864/144204)
Branch Coverage: 23.07% (19288/83612)
Coverage Report: http://coverage.selectdb-in.cc/coverage/af4283100841534a9b0736553fc9f78340dc43f1_af4283100841534a9b0736553fc9f78340dc43f1/report/index.html

xy720
xy720 previously approved these changes Apr 20, 2024
Copy link
Member

@xy720 xy720 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 20, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring previously approved these changes Apr 20, 2024
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@deardeng deardeng dismissed stale reviews from dataroaring and xy720 via 2cb3679 April 20, 2024 16:07
@deardeng
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 20, 2024
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39097 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2cb3679a64b24f6544399eea0140145d75db4b56, data reload: false

------ Round 1 ----------------------------------
q1	17590	5333	4511	4511
q2	2013	180	180	180
q3	10464	1158	1133	1133
q4	10205	722	813	722
q5	7505	2716	2611	2611
q6	218	132	133	132
q7	1014	616	591	591
q8	9234	2120	2086	2086
q9	7379	6648	6608	6608
q10	8563	3558	3493	3493
q11	462	234	228	228
q12	482	231	225	225
q13	18665	2986	2955	2955
q14	285	226	228	226
q15	505	501	476	476
q16	543	395	389	389
q17	980	732	668	668
q18	7421	6856	6809	6809
q19	6705	1536	1546	1536
q20	646	325	300	300
q21	3562	2960	2911	2911
q22	374	307	312	307
Total cold run time: 114815 ms
Total hot run time: 39097 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4337	4262	4333	4262
q2	387	274	256	256
q3	2993	2788	2770	2770
q4	1870	1572	1583	1572
q5	5378	5369	5344	5344
q6	213	125	128	125
q7	2222	1842	1873	1842
q8	3270	3383	3464	3383
q9	8631	8646	8620	8620
q10	3956	3751	3729	3729
q11	598	481	486	481
q12	749	601	593	593
q13	17026	2956	2944	2944
q14	291	272	266	266
q15	505	478	478	478
q16	485	422	427	422
q17	1781	1486	1513	1486
q18	8311	7880	8301	7880
q19	1697	1667	1625	1625
q20	2105	1835	1824	1824
q21	5911	5232	5137	5137
q22	533	485	464	464
Total cold run time: 73249 ms
Total hot run time: 55503 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.37% (8916/25205)
Line Coverage: 27.09% (73292/270523)
Region Coverage: 26.24% (37875/144357)
Branch Coverage: 23.05% (19285/83684)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2cb3679a64b24f6544399eea0140145d75db4b56_2cb3679a64b24f6544399eea0140145d75db4b56/report/index.html

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 21, 2024
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@yujun777 yujun777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit 4d574b9 into apache:master Apr 22, 2024
28 of 32 checks passed
deardeng added a commit to deardeng/incubator-doris that referenced this pull request Apr 22, 2024
yiguolei pushed a commit that referenced this pull request Apr 22, 2024
* [Fix](clean trash) Fix clean trash use agent task (#33912)

* add .h
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. meta-change reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants