Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[optimize](cooldown)Shorten the _meta_lock lock interval #27118

Merged
merged 1 commit into from
Nov 17, 2023
Merged

[optimize](cooldown)Shorten the _meta_lock lock interval #27118

merged 1 commit into from
Nov 17, 2023

Conversation

xingyingone
Copy link
Contributor

Change the two passes of _rs_version_map to one, reducing cpu overhead and shortening the lock interval of _meta_lock

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@alexxing662
[optimize](cooldown)Shorten the _meta_lock lock interval
314c4c8 
Change the two passes of _rs_version_map to one, reducing cpu overhead and shortening the lock interval of _meta_lock
@xingyingone
Copy link
Contributor Author

run buildall

alexxing662
alexxing662 approved these changes Nov 16, 2023
View reviewed changes
Copy link
Contributor

@alexxing662 alexxing662 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.75% (8410/22883)
Line Coverage: 29.28% (68416/233641)
Region Coverage: 27.86% (35352/126870)
Branch Coverage: 24.62% (18062/73356)
Coverage Report: http://coverage.selectdb-in.cc/coverage/314c4c8e5421129d2bdb25747822f92c126ada43_314c4c8e5421129d2bdb25747822f92c126ada43/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 314c4c8e5421129d2bdb25747822f92c126ada43, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4942	4671	4655	4655
q2	362	165	156	156
q3	2039	1918	1898	1898
q4	1386	1227	1228	1227
q5	3959	3959	4015	3959
q6	250	130	134	130
q7	1426	879	884	879
q8	2740	2779	2781	2779
q9	9722	9664	9591	9591
q10	3454	3523	3519	3519
q11	376	241	242	241
q12	441	303	300	300
q13	4555	3841	3764	3764
q14	323	280	279	279
q15	574	532	525	525
q16	676	579	584	579
q17	1140	959	929	929
q18	7864	7453	7425	7425
q19	1679	1703	1691	1691
q20	535	302	289	289
q21	4443	3952	4013	3952
q22	476	364	372	364
Total cold run time: 53362 ms
Total hot run time: 49131 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4594	4556	4573	4556
q2	326	223	269	223
q3	4003	3988	4005	3988
q4	2707	2686	2706	2686
q5	9597	9591	9571	9571
q6	253	123	126	123
q7	2602	2233	2264	2233
q8	4452	4498	4437	4437
q9	13128	13099	13162	13099
q10	4100	4183	4198	4183
q11	751	641	646	641
q12	977	824	817	817
q13	4542	3612	3575	3575
q14	384	352	343	343
q15	581	531	526	526
q16	732	671	665	665
q17	3915	3925	3872	3872
q18	9503	8887	9134	8887
q19	1814	1780	1769	1769
q20	2405	2049	2057	2049
q21	8805	8723	8460	8460
q22	940	793	845	793
Total cold run time: 81111 ms
Total hot run time: 77496 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.15 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17099637929 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 314c4c8e5421129d2bdb25747822f92c126ada43, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4928	4661	4644	4644
q2	357	157	158	157
q3	2022	2002	1939	1939
q4	1399	1286	1272	1272
q5	3966	3936	4014	3936
q6	245	128	128	128
q7	1374	869	881	869
q8	2769	2784	2775	2775
q9	9745	9786	9550	9550
q10	3464	3539	3515	3515
q11	383	244	241	241
q12	435	293	302	293
q13	4589	3838	3813	3813
q14	310	280	290	280
q15	572	543	528	528
q16	662	592	582	582
q17	1138	987	967	967
q18	7768	7310	7434	7310
q19	1681	1696	1686	1686
q20	572	326	313	313
q21	4387	3981	3996	3981
q22	477	370	375	370
Total cold run time: 53243 ms
Total hot run time: 49149 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4614	4618	4569	4569
q2	316	225	250	225
q3	4016	4005	3971	3971
q4	2708	2692	2699	2692
q5	9651	9625	9548	9548
q6	242	120	120	120
q7	2594	2271	2297	2271
q8	4431	4441	4483	4441
q9	13201	13065	13150	13065
q10	4072	4156	4182	4156
q11	784	758	642	642
q12	975	824	821	821
q13	4312	3575	3517	3517
q14	377	340	355	340
q15	578	524	531	524
q16	734	674	681	674
q17	3911	3888	3888	3888
q18	9555	9019	9025	9019
q19	1826	1769	1785	1769
q20	2394	2037	2045	2037
q21	8790	8574	8296	8296
q22	938	840	859	840
Total cold run time: 81019 ms
Total hot run time: 77425 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.91 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17100478872 Bytes

yiguolei
yiguolei approved these changes Nov 17, 2023
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit fdec286 into apache:master Nov 17, 2023
30 of 31 checks passed
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 17, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

superdiaodiao pushed a commit to superdiaodiao/doris that referenced this pull request Nov 21, 2023
@xingyingone @alexxing662 @superdiaodiao
[optimize](cooldown)Shorten the _meta_lock lock interval (apache#27118)
5b69acf 
Change the two passes of _rs_version_map to one, reducing cpu overhead and shortening the lock interval of _meta_lock

Co-authored-by: xingying01@corp.netease.com <xingying01@corp.netease.com>
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
@xingyingone @alexxing662 @seawinde
[optimize](cooldown)Shorten the _meta_lock lock interval (apache#27118)
c8e0a18 
Change the two passes of _rs_version_map to one, reducing cpu overhead and shortening the lock interval of _meta_lock

Co-authored-by: xingying01@corp.netease.com <xingying01@corp.netease.com>
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@xingyingone @alexxing662
[optimize](cooldown)Shorten the _meta_lock lock interval (apache#27118)
3ededd7 
Change the two passes of _rs_version_map to one, reducing cpu overhead and shortening the lock interval of _meta_lock

Co-authored-by: xingying01@corp.netease.com <xingying01@corp.netease.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiguolei yiguolei yiguolei approved these changes

@ByteYue ByteYue ByteYue approved these changes

@alexxing662 alexxing662 alexxing662 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@xingyingone @doris-robot @yiguolei @ByteYue @alexxing662