Skip to content

[fix](storage medium) Fix partition show storage medium not right when use default medium#34123

Merged
dataroaring merged 4 commits intoapache:branch-2.0from
deardeng:pick-30160-2.0
Apr 27, 2024
Merged

[fix](storage medium) Fix partition show storage medium not right when use default medium#34123
dataroaring merged 4 commits intoapache:branch-2.0from
deardeng:pick-30160-2.0

Conversation

@deardeng
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49201 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f47d92bb327998142d8150d6f28a8d0b9bbe9de, data reload: false

------ Round 1 ----------------------------------
q1	17660	4433	4302	4302
q2	2038	151	143	143
q3	10457	1864	1937	1864
q4	10298	1241	1317	1241
q5	8502	3776	3881	3776
q6	226	121	124	121
q7	2019	1603	1608	1603
q8	9307	2695	2691	2691
q9	10437	10122	10298	10122
q10	8646	3517	3495	3495
q11	428	243	244	243
q12	466	300	301	300
q13	18341	3949	4020	3949
q14	349	313	328	313
q15	501	463	454	454
q16	674	567	555	555
q17	1133	936	931	931
q18	7292	6884	6852	6852
q19	1700	1592	1500	1500
q20	534	293	294	293
q21	4390	4071	4069	4069
q22	471	384	391	384
Total cold run time: 115869 ms
Total hot run time: 49201 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4322	4274	4269	4269
q2	319	223	223	223
q3	4200	4117	4115	4115
q4	2743	2738	2734	2734
q5	7159	7058	7074	7058
q6	234	117	119	117
q7	3221	2810	2819	2810
q8	4328	4454	4475	4454
q9	16862	16720	16621	16621
q10	4199	4228	4273	4228
q11	743	701	672	672
q12	1040	860	839	839
q13	6988	3783	3755	3755
q14	446	411	416	411
q15	503	458	465	458
q16	727	682	678	678
q17	3887	3908	3847	3847
q18	8725	8816	8784	8784
q19	1715	1707	1679	1679
q20	2334	2156	2146	2146
q21	8439	8463	8374	8374
q22	1005	927	942	927
Total cold run time: 84139 ms
Total hot run time: 79199 ms

@deardeng
Copy link
Contributor Author

run buildall

Copy link
Contributor

@yujun777 yujun777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 49645 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 915b5a5d9a3990f908f05d40653080c110b69ee4, data reload: false

------ Round 1 ----------------------------------
q1	18009	4429	4321	4321
q2	2032	148	140	140
q3	10453	1877	1931	1877
q4	10292	1261	1322	1261
q5	8495	3901	3931	3901
q6	231	123	122	122
q7	1995	1626	1640	1626
q8	9299	2719	2724	2719
q9	10451	10320	10188	10188
q10	8640	3485	3530	3485
q11	416	249	250	249
q12	460	304	309	304
q13	18331	3921	4038	3921
q14	349	329	319	319
q15	500	465	458	458
q16	671	583	583	583
q17	1140	966	953	953
q18	7223	6948	6904	6904
q19	1676	1577	1585	1577
q20	513	314	298	298
q21	4502	4046	4085	4046
q22	495	393	393	393
Total cold run time: 116173 ms
Total hot run time: 49645 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4428	4350	4326	4326
q2	321	231	218	218
q3	4152	4154	4159	4154
q4	2750	2748	2766	2748
q5	7234	7066	7089	7066
q6	234	119	116	116
q7	3262	2800	2818	2800
q8	4335	4482	4452	4452
q9	17050	16727	16745	16727
q10	4253	4271	4233	4233
q11	780	711	680	680
q12	1044	856	847	847
q13	7007	3757	3732	3732
q14	466	425	438	425
q15	509	459	456	456
q16	733	686	685	685
q17	3851	3814	3864	3814
q18	8771	8872	8941	8872
q19	1740	1699	1681	1681
q20	2398	2150	2116	2116
q21	8491	8471	8469	8469
q22	1031	934	941	934
Total cold run time: 84840 ms
Total hot run time: 79551 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 202178 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 915b5a5d9a3990f908f05d40653080c110b69ee4, data reload: false

query1	944	386	373	373
query2	6517	2796	2631	2631
query3	6914	200	196	196
query4	20808	17943	17925	17925
query5	19710	6483	6441	6441
query6	288	210	235	210
query7	4164	297	312	297
query8	249	263	242	242
query9	3198	2780	2683	2683
query10	404	291	286	286
query11	11408	10727	10706	10706
query12	117	76	69	69
query13	5580	693	673	673
query14	18137	13326	13450	13326
query15	376	212	222	212
query16	6462	264	257	257
query17	1733	1460	888	888
query18	2312	406	400	400
query19	207	142	142	142
query20	82	72	74	72
query21	186	92	89	89
query22	5287	5080	5130	5080
query23	32776	32045	31777	31777
query24	6917	6529	6540	6529
query25	532	431	410	410
query26	525	162	159	159
query27	1906	302	292	292
query28	6282	2447	2397	2397
query29	2883	2837	2794	2794
query30	250	159	164	159
query31	918	717	724	717
query32	74	60	58	58
query33	398	251	238	238
query34	830	477	482	477
query35	1129	925	911	911
query36	1303	1099	1252	1099
query37	89	59	58	58
query38	3099	2901	2963	2901
query39	1367	1312	1317	1312
query40	204	99	93	93
query41	37	38	33	33
query42	85	93	82	82
query43	760	729	668	668
query44	1135	733	725	725
query45	241	225	224	224
query46	1209	964	946	946
query47	1813	1858	1641	1641
query48	1015	721	715	715
query49	611	360	361	360
query50	854	594	616	594
query51	4827	4682	4662	4662
query52	91	77	75	75
query53	458	314	324	314
query54	2642	2427	2447	2427
query55	90	77	78	77
query56	197	192	197	192
query57	1220	1127	1083	1083
query58	200	208	204	204
query59	4134	3864	4188	3864
query60	205	196	192	192
query61	86	80	84	80
query62	811	472	490	472
query63	462	332	334	332
query64	2519	1601	1414	1414
query65	3636	3566	3566	3566
query66	778	362	377	362
query67	15450	16096	14855	14855
query68	8498	645	697	645
query69	566	339	332	332
query70	1669	1471	1788	1471
query71	405	297	313	297
query72	6436	3433	3417	3417
query73	733	324	328	324
query74	6346	5917	5827	5827
query75	4740	3750	3657	3657
query76	4842	1153	1221	1153
query77	671	253	248	248
query78	12779	11457	11864	11457
query79	9052	671	644	644
query80	2168	398	392	392
query81	513	227	228	227
query82	1698	100	97	97
query83	177	131	136	131
query84	258	69	67	67
query85	1262	307	293	293
query86	345	331	279	279
query87	3289	3022	3038	3022
query88	5133	2324	2321	2321
query89	382	296	301	296
query90	1801	195	203	195
query91	164	130	134	130
query92	57	50	53	50
query93	5241	595	573	573
query94	831	208	212	208
query95	1097	1050	1060	1050
query96	657	334	323	323
query97	6467	6280	6357	6280
query98	185	171	165	165
query99	2935	869	955	869
Total cold run time: 313528 ms
Total hot run time: 202178 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 915b5a5d9a3990f908f05d40653080c110b69ee4, data reload: false

query1	0.02	0.02	0.03
query2	0.07	0.02	0.02
query3	0.25	0.04	0.05
query4	1.79	0.06	0.08
query5	0.54	0.52	0.52
query6	1.24	0.62	0.62
query7	0.01	0.01	0.01
query8	0.04	0.02	0.02
query9	0.54	0.48	0.49
query10	0.53	0.55	0.52
query11	0.11	0.08	0.08
query12	0.12	0.09	0.09
query13	0.62	0.62	0.61
query14	0.79	0.78	0.78
query15	0.79	0.76	0.77
query16	0.37	0.36	0.38
query17	1.03	0.99	1.01
query18	0.23	0.27	0.24
query19	1.91	1.84	1.86
query20	0.02	0.02	0.01
query21	15.49	0.55	0.53
query22	2.27	2.76	1.68
query23	17.15	0.97	1.08
query24	6.85	1.17	0.60
query25	0.32	0.12	0.05
query26	0.67	0.15	0.14
query27	0.05	0.04	0.04
query28	6.57	0.76	0.76
query29	12.71	2.11	2.34
query30	0.60	0.54	0.56
query31	2.80	0.39	0.37
query32	3.37	0.50	0.50
query33	3.11	3.06	3.10
query34	15.26	4.77	4.80
query35	4.85	4.84	4.82
query36	1.05	1.02	1.04
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.13	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.01
Total cold run time: 104.51 s
Total hot run time: 30.27 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 915b5a5d9a3990f908f05d40653080c110b69ee4 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

@dataroaring
Copy link
Contributor

run buildall

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 26, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 49288 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7d1845fce9ae6a36aa7deb321794eb7f90593794, data reload: false

------ Round 1 ----------------------------------
q1	17725	4372	4330	4330
q2	2032	145	140	140
q3	10422	1900	1915	1900
q4	10350	1228	1324	1228
q5	8513	3849	3922	3849
q6	230	123	121	121
q7	2061	1625	1607	1607
q8	9276	2716	2723	2716
q9	10516	10226	10077	10077
q10	8635	3499	3477	3477
q11	426	242	244	242
q12	458	308	299	299
q13	18351	3962	3956	3956
q14	355	324	323	323
q15	505	457	458	457
q16	691	561	563	561
q17	1156	946	993	946
q18	7142	6744	6911	6744
q19	1695	1562	1507	1507
q20	556	295	292	292
q21	4373	4131	4131	4131
q22	483	385	395	385
Total cold run time: 115951 ms
Total hot run time: 49288 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4313	4306	4333	4306
q2	317	221	213	213
q3	4129	4154	4149	4149
q4	2741	2755	2729	2729
q5	7145	7047	7152	7047
q6	234	117	114	114
q7	3253	2798	2859	2798
q8	4329	4390	4419	4390
q9	16881	16800	16637	16637
q10	4234	4290	4289	4289
q11	740	680	645	645
q12	1029	856	850	850
q13	7639	3740	3733	3733
q14	446	424	429	424
q15	490	462	450	450
q16	735	685	689	685
q17	3782	3841	3843	3841
q18	8707	8710	8843	8710
q19	1712	1695	1630	1630
q20	2329	2093	2074	2074
q21	8392	8268	8434	8268
q22	1007	979	922	922
Total cold run time: 84584 ms
Total hot run time: 78904 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 201530 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7d1845fce9ae6a36aa7deb321794eb7f90593794, data reload: false

query1	914	381	373	373
query2	6546	2795	2440	2440
query3	6915	204	197	197
query4	21316	17978	18088	17978
query5	19720	6473	6465	6465
query6	291	216	229	216
query7	4171	292	291	291
query8	275	234	244	234
query9	3156	2712	2652	2652
query10	419	284	307	284
query11	11339	10598	10526	10526
query12	120	75	74	74
query13	5575	681	662	662
query14	17150	13583	13203	13203
query15	360	221	222	221
query16	6464	274	250	250
query17	1568	1448	872	872
query18	2282	400	399	399
query19	209	147	149	147
query20	77	73	73	73
query21	187	99	89	89
query22	5193	5030	5009	5009
query23	32517	31994	31832	31832
query24	7042	6520	6491	6491
query25	494	416	415	415
query26	519	157	151	151
query27	1911	293	286	286
query28	6172	2402	2375	2375
query29	2843	2671	2829	2671
query30	237	160	160	160
query31	906	733	744	733
query32	66	61	59	59
query33	392	247	248	247
query34	831	457	467	457
query35	1130	861	874	861
query36	1365	1095	1197	1095
query37	86	64	61	61
query38	3154	2919	2929	2919
query39	1356	1335	1341	1335
query40	198	91	98	91
query41	36	34	34	34
query42	86	88	104	88
query43	942	729	706	706
query44	1122	723	714	714
query45	244	228	225	225
query46	1227	945	955	945
query47	1718	1708	1863	1708
query48	1015	731	703	703
query49	629	364	382	364
query50	869	574	571	571
query51	4701	4627	4616	4616
query52	84	85	75	75
query53	450	313	312	312
query54	2644	2423	2485	2423
query55	92	71	79	71
query56	204	201	218	201
query57	1188	1098	1020	1020
query58	207	200	201	200
query59	4058	4119	3880	3880
query60	199	192	199	192
query61	90	84	82	82
query62	796	483	486	483
query63	467	337	328	328
query64	4157	1643	1467	1467
query65	3600	3525	3541	3525
query66	640	362	384	362
query67	15641	15348	14782	14782
query68	9517	651	679	651
query69	576	337	337	337
query70	1738	1426	1492	1426
query71	414	301	308	301
query72	6479	3419	3445	3419
query73	731	315	317	315
query74	6389	5785	5848	5785
query75	5396	3717	3741	3717
query76	5902	1184	1239	1184
query77	971	248	246	246
query78	12469	11650	11801	11650
query79	8005	642	673	642
query80	1145	394	387	387
query81	486	239	225	225
query82	1621	97	91	91
query83	158	137	128	128
query84	259	69	67	67
query85	846	302	299	299
query86	324	306	283	283
query87	3225	3043	3034	3034
query88	4595	2328	2312	2312
query89	384	303	292	292
query90	1955	208	206	206
query91	163	135	142	135
query92	57	50	54	50
query93	6275	558	549	549
query94	667	206	199	199
query95	1129	1077	1067	1067
query96	662	322	326	322
query97	6401	6320	6222	6222
query98	188	176	172	172
query99	2970	846	963	846
Total cold run time: 314669 ms
Total hot run time: 201530 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.95 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7d1845fce9ae6a36aa7deb321794eb7f90593794, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.03	0.02
query3	0.24	0.04	0.05
query4	1.80	0.06	0.06
query5	0.53	0.52	0.53
query6	1.22	0.62	0.61
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.48	0.48
query10	0.54	0.53	0.52
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.61	0.61	0.63
query14	0.78	0.80	0.79
query15	0.78	0.75	0.76
query16	0.37	0.38	0.36
query17	1.02	1.01	1.00
query18	0.19	0.29	0.23
query19	1.93	1.83	1.86
query20	0.02	0.01	0.02
query21	15.45	0.55	0.53
query22	2.08	2.20	2.00
query23	17.00	0.99	1.06
query24	5.86	1.93	1.81
query25	0.39	0.13	0.06
query26	0.73	0.15	0.14
query27	0.05	0.04	0.03
query28	5.61	0.73	0.75
query29	12.76	2.37	2.27
query30	0.56	0.54	0.52
query31	2.80	0.39	0.39
query32	3.37	0.50	0.50
query33	3.09	3.07	3.07
query34	15.29	4.78	4.78
query35	4.83	4.83	4.86
query36	1.06	1.01	1.01
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 102.24 s
Total hot run time: 31.95 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7d1845fce9ae6a36aa7deb321794eb7f90593794 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@dataroaring dataroaring merged commit 7a15962 into apache:branch-2.0 Apr 27, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
…n use default medium (apache#34123)

Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. kind/test reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants