Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhance](cooldown) Reduce the locking interval for cooldown task #26984

Merged
merged 1 commit into from
Nov 16, 2023
Merged

[enhance](cooldown) Reduce the locking interval for cooldown task #26984

merged 1 commit into from
Nov 16, 2023

Conversation

xingyingone
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

alexxing662
alexxing662 approved these changes Nov 14, 2023
View reviewed changes
Copy link
Contributor

@alexxing662 alexxing662 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@xingyingone
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Nov 14, 2023
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/olap_server.cpp Outdated
bool is_skip =
cur_time - tablet->last_failed_follow_cooldown_time() < skip_failed_interval ||
TABLET_RUNNING != tablet->tablet_state();
if (is_skip) return is_skip;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: statement should be inside braces [readability-braces-around-statements]

Suggested change
if (is_skip) return is_skip;
if (is_skip) { return is_skip;
}

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17097010013 Bytes

@xingyingone
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.74% (8420/22915)
Line Coverage: 29.27% (68413/233719)
Region Coverage: 27.87% (35364/126867)
Branch Coverage: 24.67% (18076/73280)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3ae05d483c06bb1f220cac5b0a47d43fbec624eb_3ae05d483c06bb1f220cac5b0a47d43fbec624eb/report/index.html

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.74% (8419/22918)
Line Coverage: 29.27% (68413/233732)
Region Coverage: 27.87% (35358/126875)
Branch Coverage: 24.66% (18072/73292)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1eee59e0e10fd79be33c04d9112a7614e2afc919_1eee59e0e10fd79be33c04d9112a7614e2afc919/report/index.html

@xingyingone
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.80% (8419/22879)
Line Coverage: 29.29% (68416/233616)
Region Coverage: 27.89% (35364/126817)
Branch Coverage: 24.66% (18076/73294)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1eee59e0e10fd79be33c04d9112a7614e2afc919_1eee59e0e10fd79be33c04d9112a7614e2afc919/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 1eee59e0e10fd79be33c04d9112a7614e2afc919, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5250	5081	5091	5081
q2	356	164	158	158
q3	2035	2038	1966	1966
q4	1412	1358	1366	1358
q5	3982	3945	4001	3945
q6	260	129	128	128
q7	1445	896	900	896
q8	2769	2782	2766	2766
q9	9748	9566	9569	9566
q10	3448	3566	3538	3538
q11	373	256	257	256
q12	425	278	278	278
q13	4544	4147	4128	4128
q14	321	305	292	292
q15	646	554	539	539
q16	671	579	576	576
q17	1140	1086	1084	1084
q18	8104	7504	7560	7504
q19	1669	1671	1678	1671
q20	562	305	286	286
q21	4688	4354	4354	4354
q22	501	401	418	401
Total cold run time: 54349 ms
Total hot run time: 50771 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5020	5014	5038	5014
q2	332	230	228	228
q3	4036	4007	4019	4007
q4	2777	2745	2763	2745
q5	9686	9553	9594	9553
q6	244	123	117	117
q7	3073	2472	2485	2472
q8	4824	4806	4819	4806
q9	13278	13057	13182	13057
q10	4052	4165	4199	4165
q11	733	665	650	650
q12	993	851	819	819
q13	4248	3869	3899	3869
q14	371	340	352	340
q15	585	548	565	548
q16	780	669	710	669
q17	3888	3913	3995	3913
q18	9690	9376	9360	9360
q19	1833	1778	1764	1764
q20	2402	2055	2028	2028
q21	8768	8652	8685	8652
q22	972	866	884	866
Total cold run time: 82585 ms
Total hot run time: 79642 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.94 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17097082227 Bytes

@xingyingone
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 1eee59e0e10fd79be33c04d9112a7614e2afc919, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5228	5148	5076	5076
q2	352	164	157	157
q3	2038	2037	2031	2031
q4	1403	1375	1351	1351
q5	3953	3941	4037	3941
q6	254	133	132	132
q7	1470	879	922	879
q8	2777	2784	2764	2764
q9	9802	9592	9579	9579
q10	3439	3551	3542	3542
q11	385	260	257	257
q12	432	283	282	282
q13	4568	4147	4133	4133
q14	320	294	283	283
q15	633	554	564	554
q16	674	580	584	580
q17	1136	1085	1086	1085
q18	8122	7661	7736	7661
q19	1686	1679	1703	1679
q20	583	295	293	293
q21	4688	4346	4389	4346
q22	523	388	412	388
Total cold run time: 54466 ms
Total hot run time: 50993 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5004	5007	5026	5007
q2	366	219	246	219
q3	4144	4034	4015	4015
q4	2789	2742	2755	2742
q5	9672	9680	9841	9680
q6	249	122	128	122
q7	3067	2536	2482	2482
q8	4841	4803	4862	4803
q9	13359	13176	13234	13176
q10	4098	4172	4174	4172
q11	739	634	621	621
q12	982	823	850	823
q13	4263	3848	3866	3848
q14	372	359	360	359
q15	577	537	559	537
q16	749	692	727	692
q17	3907	3946	3931	3931
q18	9666	9402	9174	9174
q19	1929	1776	1767	1767
q20	2372	2044	2057	2044
q21	8848	8863	8717	8717
q22	949	835	807	807
Total cold run time: 82942 ms
Total hot run time: 79738 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44 seconds
stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17096608403 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.79% (8418/22879)
Line Coverage: 29.28% (68392/233615)
Region Coverage: 27.87% (35347/126817)
Branch Coverage: 24.65% (18065/73294)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1eee59e0e10fd79be33c04d9112a7614e2afc919_1eee59e0e10fd79be33c04d9112a7614e2afc919/report/index.html

yiguolei
yiguolei approved these changes Nov 16, 2023
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 16, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@yiguolei yiguolei merged commit b8b86a7 into apache:master Nov 16, 2023
27 of 28 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
@xingyingone @alexxing662 @seawinde
[enhance](cooldown) Reduce the locking interval for cooldown task (ap…
bace866 
…ache#26984)

Co-authored-by: xingying01 <xingying01@corp.netease.com>
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@xingyingone @alexxing662
[enhance](cooldown) Reduce the locking interval for cooldown task (ap…
87b4ff2 
…ache#26984)

Co-authored-by: xingying01 <xingying01@corp.netease.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions left review comments

@yiguolei yiguolei yiguolei approved these changes

@platoneko platoneko platoneko approved these changes

@alexxing662 alexxing662 alexxing662 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@xingyingone @doris-robot @yiguolei @platoneko @alexxing662