Skip to content

[fix](create tablet) fix disk usage may change during chose disks #30508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jan 30, 2024
Merged

[fix](create tablet) fix disk usage may change during chose disks #30508

dataroaring merged 1 commit into from
Jan 30, 2024

Conversation

@yujun777
Copy link
Contributor

@yujun777 yujun777 commented Jan 29, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions
Copy link
Contributor

github-actions bot commented Jan 29, 2024

clang-tidy review says "All clean, LGTM! 👍"

dataroaring
dataroaring approved these changes Jan 29, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 29, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jan 29, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 29, 2024

PR approved by anyone and no changes requested.

@dataroaring
Copy link
Contributor

dataroaring commented Jan 29, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jan 29, 2024

TeamCity be ut coverage result:
Function Coverage: 36.18% (8620/23823)
Line Coverage: 28.25% (70517/249589)
Region Coverage: 27.25% (36381/133526)
Branch Coverage: 24.04% (18639/77542)
Coverage Report: http://coverage.selectdb-in.cc/coverage/74e502289786863f5a1cf43e616830815a3cb167_74e502289786863f5a1cf43e616830815a3cb167/report/index.html

@doris-robot
Copy link

doris-robot commented Jan 29, 2024

TPC-H: Total hot run time: 38337 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 74e502289786863f5a1cf43e616830815a3cb167, data reload: false

------ Round 1 ----------------------------------
q1	18260	5051	4927	4927
q2	2048	138	130	130
q3	10716	927	949	927
q4	4718	755	717	717
q5	8187	2827	2901	2827
q6	188	119	119	119
q7	1149	745	741	741
q8	9478	1997	2012	1997
q9	7235	6372	6368	6368
q10	8348	2632	2588	2588
q11	423	216	198	198
q12	803	301	302	301
q13	18989	3592	3546	3546
q14	281	295	288	288
q15	535	542	555	542
q16	898	421	408	408
q17	1377	529	508	508
q18	8208	6145	6163	6145
q19	1698	1389	1373	1373
q20	635	349	324	324
q21	6881	3120	3069	3069
q22	814	294	302	294
Total cold run time: 111869 ms
Total hot run time: 38337 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5172	4850	4979	4850
q2	318	252	246	246
q3	2967	2805	2696	2696
q4	1873	1668	1601	1601
q5	5352	5252	5197	5197
q6	211	115	117	115
q7	2150	1845	1773	1773
q8	3185	3219	3217	3217
q9	8693	8669	18059	8669
q10	5463	3427	3516	3427
q11	549	486	471	471
q12	763	585	585	585
q13	16945	3097	3064	3064
q14	280	251	272	251
q15	537	490	488	488
q16	510	453	463	453
q17	2005	1654	1713	1654
q18	8751	9355	8115	8115
q19	23936	1558	1508	1508
q20	4688	1902	1907	1902
q21	15091	4866	4905	4866
q22	985	461	489	461
Total cold run time: 110424 ms
Total hot run time: 55609 ms

@doris-robot
Copy link

doris-robot commented Jan 29, 2024

TPC-DS: Total hot run time: 172909 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 74e502289786863f5a1cf43e616830815a3cb167, data reload: false

query1	961	335	331	331
query2	6532	2005	1963	1963
query3	6699	199	195	195
query4	32473	22357	22041	22041
query5	4452	421	408	408
query6	244	160	152	152
query7	4607	263	261	261
query8	255	171	175	171
query9	8614	2262	2259	2259
query10	412	201	203	201
query11	17699	15310	15309	15309
query12	119	64	62	62
query13	1679	365	376	365
query14	9227	6498	6469	6469
query15	206	179	185	179
query16	5697	253	244	244
query17	953	481	463	463
query18	1815	256	251	251
query19	170	135	129	129
query20	76	73	66	66
query21	206	140	128	128
query22	4852	4808	4488	4488
query23	31080	30340	30362	30340
query24	10035	2778	2784	2778
query25	535	306	310	306
query26	877	145	140	140
query27	3025	286	281	281
query28	6088	1837	1831	1831
query29	933	620	607	607
query30	281	135	138	135
query31	902	695	723	695
query32	75	49	51	49
query33	582	220	209	209
query34	843	458	453	453
query35	817	767	743	743
query36	1303	1214	1210	1210
query37	90	56	57	56
query38	3279	3228	3158	3158
query39	1317	1240	1243	1240
query40	194	84	87	84
query41	41	36	34	34
query42	85	82	79	79
query43	520	488	494	488
query44	1038	703	702	702
query45	189	179	176	176
query46	1042	653	658	653
query47	1557	1499	1524	1499
query48	399	302	300	300
query49	1092	291	285	285
query50	671	309	319	309
query51	5293	5147	5140	5140
query52	93	78	74	74
query53	334	264	264	264
query54	209	182	188	182
query55	78	75	73	73
query56	174	160	172	160
query57	989	893	922	893
query58	173	152	157	152
query59	2777	2656	2538	2538
query60	213	181	175	175
query61	85	87	79	79
query62	620	345	354	345
query63	288	265	269	265
query64	4020	1731	1731	1731
query65	3302	3231	3202	3202
query66	1176	312	307	307
query67	14252	14107	14322	14107
query68	3729	504	493	493
query69	432	307	296	296
query70	1572	1535	1498	1498
query71	262	216	228	216
query72	3727	2805	2818	2805
query73	684	309	317	309
query74	6591	6301	6260	6260
query75	2835	2338	2289	2289
query76	2570	1008	932	932
query77	354	231	229	229
query78	9080	8860	8510	8510
query79	2809	501	480	480
query80	1955	321	312	312
query81	503	194	200	194
query82	687	84	78	78
query83	225	110	114	110
query84	295	79	77	77
query85	1983	329	330	329
query86	514	392	378	378
query87	3400	3267	3262	3262
query88	4140	2187	2177	2177
query89	416	373	352	352
query90	1901	191	184	184
query91	149	115	118	115
query92	50	45	46	45
query93	5132	446	413	413
query94	1275	155	158	155
query95	514	460	459	459
query96	609	326	314	314
query97	4237	4124	4125	4124
query98	209	194	195	194
query99	995	681	658	658
Total cold run time: 274748 ms
Total hot run time: 172909 ms

@doris-robot
Copy link

doris-robot commented Jan 29, 2024

ClickBench: Total hot run time: 31.29 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 74e502289786863f5a1cf43e616830815a3cb167, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.68	0.10	0.10
query5	0.53	0.51	0.52
query6	1.18	0.61	0.65
query7	0.02	0.01	0.01
query8	0.03	0.02	0.03
query9	0.55	0.51	0.49
query10	0.57	0.55	0.56
query11	0.12	0.08	0.09
query12	0.11	0.08	0.09
query13	0.60	0.61	0.60
query14	0.79	0.79	0.80
query15	0.79	0.77	0.77
query16	0.38	0.38	0.38
query17	0.97	1.02	1.01
query18	0.23	0.28	0.22
query19	1.90	1.82	1.81
query20	0.02	0.01	0.01
query21	15.41	0.60	0.58
query22	2.54	2.54	2.83
query23	17.07	0.95	0.96
query24	2.32	1.19	0.46
query25	0.35	0.20	0.08
query26	0.49	0.15	0.14
query27	0.05	0.05	0.04
query28	11.70	0.86	0.84
query29	12.55	3.12	3.26
query30	0.63	0.55	0.53
query31	2.79	0.36	0.35
query32	3.33	0.48	0.48
query33	3.21	3.23	3.21
query34	15.78	4.35	4.27
query35	4.40	4.32	4.24
query36	1.09	1.06	1.09
query37	0.06	0.04	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.12	0.14
query41	0.08	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.03
Total cold run time: 104.9 s
Total hot run time: 31.29 s

@doris-robot
Copy link

doris-robot commented Jan 29, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 74e502289786863f5a1cf43e616830815a3cb167 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

@dataroaring dataroaring merged commit 57ff037 into apache:master Jan 30, 2024
yujun777 added a commit to yujun777/doris that referenced this pull request Apr 23, 2024
@yujun777
[fix](create tablet) fix disk usage may change when chose disks (apac…
161311a 
…he#30508)
@xiaokang xiaokang mentioned this pull request Apr 30, 2024
Merged
yujun777 added a commit to yujun777/doris that referenced this pull request Apr 30, 2024
@yujun777
[fix](create tablet) fix disk usage may change when chose disks (apac…
1218434 
…he#30508)
xiaokang pushed a commit that referenced this pull request May 1, 2024
@yujun777
[branch-2.0](create tablet) create tablet chose disk round robin #29818
2d6b54e 
#30354 #30508 (#34034)
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@yujun777
[branch-2.0](create tablet) create tablet chose disk round robin apac…
5e698b9 
…he#29818 apache#30354 apache#30508 (apache#34034)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.10-merged dev/2.1.0 reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@yujun777 @dataroaring @doris-robot @gavinchou @xiaokang