Skip to content

[improvement](statistics)Optimize drop stats operation.#30144

Merged
morrySnow merged 1 commit intoapache:masterfrom
Jibing-Li:dropstats
Jan 22, 2024
Merged

[improvement](statistics)Optimize drop stats operation.#30144
morrySnow merged 1 commit intoapache:masterfrom
Jibing-Li:dropstats

Conversation

@Jibing-Li
Copy link
Contributor

Before, drop stats operation need to call columns * followers times of isMaster() function and the same times of rpc to drop remote column stats. This pr is to reduce the rpc calls and use more efficient way to check master node instead of using isMaster()

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@Jibing-Li Jibing-Li marked this pull request as ready for review January 19, 2024 08:59
@Jibing-Li
Copy link
Contributor Author

run buildall

@wm1581066 wm1581066 added the usercase Important user case type label label Jan 19, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 39200 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 94cb81318a5c9b692962e41a10a4c62f2818f151, data reload: false

------ Round 1 ----------------------------------
q1	17612	5247	5298	5247
q2	2045	150	132	132
q3	10645	1154	1154	1154
q4	10774	812	857	812
q5	7887	3217	3173	3173
q6	202	125	121	121
q7	876	521	482	482
q8	9386	1928	1968	1928
q9	7676	6506	6488	6488
q10	8450	3022	3046	3022
q11	420	235	215	215
q12	351	189	188	188
q13	17985	3372	3352	3352
q14	238	217	216	216
q15	550	513	495	495
q16	425	365	371	365
q17	935	545	542	542
q18	7651	6922	6948	6922
q19	1553	1387	1370	1370
q20	608	311	294	294
q21	2735	2366	2390	2366
q22	365	316	318	316
Total cold run time: 109369 ms
Total hot run time: 39200 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5459	5233	5296	5233
q2	329	212	225	212
q3	3329	3236	3213	3213
q4	2085	2063	2059	2059
q5	5933	5938	5929	5929
q6	196	115	127	115
q7	2299	1837	1901	1837
q8	3198	3363	3374	3363
q9	8964	8701	8904	8701
q10	3930	3782	3813	3782
q11	571	455	451	451
q12	803	604	634	604
q13	16932	3144	3111	3111
q14	294	263	264	263
q15	555	504	500	500
q16	509	455	465	455
q17	1880	1837	1849	1837
q18	9504	10030	9569	9569
q19	20615	1585	1517	1517
q20	4603	1917	1914	1914
q21	14269	5299	5347	5299
q22	966	576	559	559
Total cold run time: 107223 ms
Total hot run time: 60523 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 176178 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 94cb81318a5c9b692962e41a10a4c62f2818f151, data reload: false

query1	947	328	324	324
query2	6553	2034	1810	1810
query3	6692	200	200	200
query4	32008	22304	22238	22238
query5	6898	599	595	595
query6	245	168	154	154
query7	4600	263	255	255
query8	226	170	177	170
query9	8371	2524	2520	2520
query10	417	232	240	232
query11	16966	15569	15446	15446
query12	117	68	69	68
query13	1710	375	392	375
query14	10523	6990	6909	6909
query15	208	177	184	177
query16	4616	234	226	226
query17	992	466	474	466
query18	1798	258	247	247
query19	171	135	131	131
query20	72	72	64	64
query21	198	136	123	123
query22	4938	4803	4674	4674
query23	31468	30783	30595	30595
query24	12646	2775	2828	2775
query25	571	305	300	300
query26	1815	141	141	141
query27	3223	277	271	271
query28	7345	1835	1815	1815
query29	2143	630	626	626
query30	284	135	137	135
query31	958	717	749	717
query32	73	52	50	50
query33	710	211	214	211
query34	1164	457	482	457
query35	890	734	744	734
query36	1364	1265	1236	1236
query37	174	58	61	58
query38	3358	3250	3202	3202
query39	1285	1252	1245	1245
query40	343	91	83	83
query41	38	37	36	36
query42	93	81	84	81
query43	520	503	479	479
query44	1075	679	687	679
query45	193	176	171	171
query46	1060	624	646	624
query47	1610	1560	1572	1560
query48	398	310	311	310
query49	1220	289	284	284
query50	668	302	307	302
query51	5302	5190	5144	5144
query52	85	84	76	76
query53	320	259	245	245
query54	451	291	281	281
query55	82	77	76	76
query56	182	170	170	170
query57	995	885	909	885
query58	184	162	162	162
query59	2888	2728	2713	2713
query60	201	192	181	181
query61	81	80	81	80
query62	617	373	366	366
query63	274	254	258	254
query64	6145	1751	1774	1751
query65	3331	3250	3236	3236
query66	1427	322	316	316
query67	15860	15265	15220	15220
query68	12410	501	498	498
query69	583	305	296	296
query70	1731	1531	1570	1531
query71	485	211	208	208
query72	4848	2838	2801	2801
query73	2671	311	315	311
query74	7085	6482	6288	6288
query75	4774	2335	2343	2335
query76	6228	985	1009	985
query77	642	234	231	231
query78	8990	8907	8567	8567
query79	2343	489	498	489
query80	547	314	318	314
query81	448	201	202	201
query82	211	80	84	80
query83	139	114	121	114
query84	277	66	79	66
query85	1048	322	320	320
query86	405	377	374	374
query87	3504	3319	3229	3229
query88	3282	2177	2184	2177
query89	463	346	354	346
query90	1950	183	185	183
query91	156	130	124	124
query92	52	47	43	43
query93	2709	438	427	427
query94	1213	160	155	155
query95	506	448	429	429
query96	604	317	314	314
query97	4219	4169	4140	4140
query98	213	200	179	179
query99	1016	686	712	686
Total cold run time: 300132 ms
Total hot run time: 176178 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.25 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 94cb81318a5c9b692962e41a10a4c62f2818f151, data reload: false

query1	0.03	0.02	0.02
query2	0.05	0.02	0.02
query3	0.23	0.05	0.05
query4	1.71	0.07	0.08
query5	0.55	0.52	0.53
query6	1.23	0.63	0.63
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.55	0.50	0.49
query10	0.57	0.55	0.57
query11	0.11	0.09	0.09
query12	0.11	0.09	0.09
query13	0.62	0.61	0.60
query14	0.79	0.79	0.82
query15	0.80	0.78	0.78
query16	0.39	0.40	0.38
query17	1.00	1.02	1.02
query18	0.25	0.24	0.25
query19	1.87	1.81	1.79
query20	0.01	0.02	0.02
query21	15.42	0.57	0.58
query22	2.83	2.51	1.35
query23	17.06	0.84	0.79
query24	2.54	0.81	0.91
query25	0.44	0.21	0.10
query26	0.44	0.13	0.13
query27	0.06	0.06	0.05
query28	11.85	0.76	0.76
query29	12.49	3.17	3.15
query30	0.54	0.50	0.48
query31	2.77	0.35	0.35
query32	3.38	0.48	0.48
query33	3.23	3.21	3.23
query34	15.77	4.33	4.25
query35	4.30	4.30	4.25
query36	1.10	1.08	1.08
query37	0.06	0.06	0.06
query38	0.04	0.03	0.03
query39	0.02	0.01	0.01
query40	0.16	0.12	0.14
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.55 s
Total hot run time: 30.25 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 94cb81318a5c9b692962e41a10a4c62f2818f151 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.0 seconds inserted 10000000 Rows, about 714K ops/s

@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38562 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5e9dd6cce1a35aedc83c03c5957f43401968e2af, data reload: false

------ Round 1 ----------------------------------
q1	17693	5179	5345	5179
q2	2031	149	128	128
q3	10631	1148	1123	1123
q4	10228	763	792	763
q5	7749	3189	3119	3119
q6	194	118	117	117
q7	863	493	477	477
q8	9236	1910	1941	1910
q9	7246	6384	6337	6337
q10	8195	3048	3055	3048
q11	420	202	205	202
q12	354	188	187	187
q13	17984	3330	3351	3330
q14	252	222	205	205
q15	547	510	496	496
q16	463	401	377	377
q17	934	516	481	481
q18	7407	6870	6760	6760
q19	1543	1431	1382	1382
q20	598	311	283	283
q21	2709	2356	2469	2356
q22	363	302	304	302
Total cold run time: 107640 ms
Total hot run time: 38562 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5267	5343	5342	5342
q2	326	218	211	211
q3	3336	3204	3217	3204
q4	2098	2036	2025	2025
q5	6006	5953	5967	5953
q6	198	115	116	115
q7	2258	1908	1903	1903
q8	3227	3364	3398	3364
q9	8818	8765	8720	8720
q10	3962	3826	3830	3826
q11	556	459	470	459
q12	804	615	590	590
q13	16900	3177	3151	3151
q14	278	256	261	256
q15	562	507	499	499
q16	521	482	480	480
q17	1868	1824	1864	1824
q18	9649	16604	9804	9804
q19	23963	1544	1499	1499
q20	4607	1932	1913	1913
q21	14515	5545	5437	5437
q22	978	536	572	536
Total cold run time: 110697 ms
Total hot run time: 61111 ms

morningman
morningman previously approved these changes Jan 21, 2024
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 21, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-DS: Total hot run time: 176354 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5e9dd6cce1a35aedc83c03c5957f43401968e2af, data reload: false

query1	918	329	325	325
query2	6572	1817	1809	1809
query3	6698	199	197	197
query4	34102	22235	22086	22086
query5	6915	604	585	585
query6	249	150	149	149
query7	4599	263	251	251
query8	228	187	177	177
query9	8458	2551	2512	2512
query10	424	231	219	219
query11	16343	15742	15449	15449
query12	118	69	66	66
query13	1682	375	396	375
query14	10548	6870	6858	6858
query15	216	174	186	174
query16	4606	237	226	226
query17	987	468	456	456
query18	1813	274	259	259
query19	180	134	138	134
query20	72	69	63	63
query21	180	134	130	130
query22	5155	4980	4847	4847
query23	31741	30779	30848	30779
query24	11621	2817	2810	2810
query25	560	303	306	303
query26	1616	140	140	140
query27	3224	269	273	269
query28	7131	1814	1799	1799
query29	1678	627	607	607
query30	282	139	140	139
query31	909	723	755	723
query32	77	49	48	48
query33	705	209	204	204
query34	1163	458	460	458
query35	888	719	761	719
query36	1290	1192	1204	1192
query37	94	59	60	59
query38	3324	3201	3220	3201
query39	1308	1286	1268	1268
query40	337	86	84	84
query41	38	35	34	34
query42	97	81	86	81
query43	525	483	441	441
query44	1099	674	685	674
query45	191	182	171	171
query46	1067	644	649	644
query47	1691	1601	1560	1560
query48	398	316	311	311
query49	1130	283	285	283
query50	692	299	307	299
query51	5358	5319	5253	5253
query52	97	75	77	75
query53	318	255	248	248
query54	460	287	286	286
query55	77	78	73	73
query56	179	164	166	164
query57	1010	881	969	881
query58	182	165	160	160
query59	2896	2542	2727	2542
query60	202	188	185	185
query61	81	81	82	81
query62	647	373	350	350
query63	284	260	264	260
query64	5666	1725	1755	1725
query65	3320	3251	3221	3221
query66	1250	313	315	313
query67	15974	15370	15387	15370
query68	10600	525	493	493
query69	597	307	294	294
query70	1712	1586	1493	1493
query71	519	210	209	209
query72	4662	2792	2825	2792
query73	2126	314	312	312
query74	7102	6333	6418	6333
query75	4725	2335	2250	2250
query76	6628	1019	960	960
query77	654	225	239	225
query78	9052	8727	8742	8727
query79	1129	488	502	488
query80	568	310	304	304
query81	445	207	200	200
query82	208	78	76	76
query83	138	117	116	116
query84	284	69	66	66
query85	1077	308	324	308
query86	392	397	382	382
query87	3497	3357	3314	3314
query88	3549	2183	2181	2181
query89	427	363	342	342
query90	1996	181	179	179
query91	150	127	133	127
query92	53	44	40	40
query93	3400	439	427	427
query94	1192	151	149	149
query95	506	447	436	436
query96	619	319	312	312
query97	4261	4097	4151	4097
query98	195	195	191	191
query99	1023	723	697	697
Total cold run time: 297426 ms
Total hot run time: 176354 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5e9dd6cce1a35aedc83c03c5957f43401968e2af, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.02	0.03
query3	0.22	0.05	0.04
query4	1.70	0.07	0.07
query5	0.53	0.52	0.52
query6	1.34	0.64	0.63
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.55	0.51	0.49
query10	0.55	0.56	0.56
query11	0.12	0.09	0.09
query12	0.11	0.10	0.08
query13	0.60	0.60	0.60
query14	0.79	0.81	0.80
query15	0.78	0.77	0.76
query16	0.40	0.36	0.37
query17	1.01	1.01	0.96
query18	0.25	0.22	0.25
query19	1.83	1.75	1.76
query20	0.02	0.01	0.01
query21	15.40	0.58	0.57
query22	2.50	2.72	1.67
query23	17.38	0.83	0.98
query24	2.33	0.72	1.37
query25	0.40	0.23	0.12
query26	0.55	0.13	0.14
query27	0.05	0.06	0.04
query28	11.40	0.78	0.77
query29	12.48	3.10	3.11
query30	0.56	0.50	0.55
query31	2.79	0.34	0.34
query32	3.40	0.49	0.49
query33	3.22	3.21	3.24
query34	15.81	4.28	4.24
query35	4.32	4.29	4.22
query36	1.10	1.07	1.07
query37	0.06	0.06	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.13	0.13
query41	0.08	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 105.05 s
Total hot run time: 30.27 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 5e9dd6cce1a35aedc83c03c5957f43401968e2af with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

@Jibing-Li
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 22, 2024
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 22, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 38662 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c1a745e978e3d6730638d60df4fc275641b7944b, data reload: false

------ Round 1 ----------------------------------
q1	17649	5236	5336	5236
q2	2031	141	130	130
q3	10654	1169	1150	1150
q4	10221	772	828	772
q5	7773	3153	3097	3097
q6	196	118	122	118
q7	849	494	470	470
q8	9207	1932	1941	1932
q9	7234	6362	6310	6310
q10	8165	3015	3048	3015
q11	413	210	207	207
q12	354	186	188	186
q13	17987	3337	3339	3337
q14	238	224	218	218
q15	554	495	506	495
q16	427	365	394	365
q17	933	499	573	499
q18	7491	6702	6745	6702
q19	1538	1389	1410	1389
q20	590	317	302	302
q21	2767	2430	2435	2430
q22	370	302	319	302
Total cold run time: 107641 ms
Total hot run time: 38662 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5452	5247	5181	5181
q2	325	219	210	210
q3	3362	3217	3198	3198
q4	2061	2042	2066	2042
q5	6007	5973	5878	5878
q6	215	116	117	116
q7	2326	1929	1894	1894
q8	3197	3358	3380	3358
q9	8994	8860	8745	8745
q10	3880	3766	3877	3766
q11	547	467	453	453
q12	751	648	637	637
q13	16893	3165	3139	3139
q14	303	254	261	254
q15	557	512	509	509
q16	516	467	470	467
q17	1892	1867	1881	1867
q18	9566	9402	14753	9402
q19	23694	1599	1536	1536
q20	4572	1928	1923	1923
q21	13253	5445	5394	5394
q22	976	536	510	510
Total cold run time: 109339 ms
Total hot run time: 60479 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 176724 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c1a745e978e3d6730638d60df4fc275641b7944b, data reload: false

query1	964	333	332	332
query2	6538	2144	2057	2057
query3	6699	204	205	204
query4	34453	22267	22152	22152
query5	4453	374	438	374
query6	243	155	153	153
query7	4602	258	267	258
query8	228	178	178	178
query9	8359	2513	2519	2513
query10	405	253	222	222
query11	17041	15441	15590	15441
query12	119	66	68	66
query13	1673	380	381	380
query14	10551	6974	7048	6974
query15	220	179	182	179
query16	5787	262	248	248
query17	944	482	484	482
query18	1788	255	257	255
query19	176	131	127	127
query20	70	68	70	68
query21	179	126	115	115
query22	4846	4879	4820	4820
query23	31554	30871	30706	30706
query24	11762	2836	2884	2836
query25	581	306	316	306
query26	1638	146	141	141
query27	3173	283	288	283
query28	7150	1821	1788	1788
query29	1640	631	621	621
query30	281	138	137	137
query31	944	776	746	746
query32	70	55	55	55
query33	700	213	216	213
query34	1114	464	466	464
query35	882	765	777	765
query36	1326	1220	1154	1154
query37	91	60	61	60
query38	3392	3256	3262	3256
query39	1326	1251	1287	1251
query40	330	83	85	83
query41	38	36	35	35
query42	89	85	81	81
query43	517	492	494	492
query44	1050	696	686	686
query45	192	177	177	177
query46	1068	657	653	653
query47	1667	1472	1562	1472
query48	380	310	319	310
query49	1144	296	281	281
query50	690	316	310	310
query51	5382	5169	5209	5169
query52	92	75	81	75
query53	319	256	261	256
query54	233	190	189	189
query55	85	72	73	72
query56	177	171	169	169
query57	1046	925	885	885
query58	183	160	157	157
query59	3007	2702	2734	2702
query60	216	186	193	186
query61	86	84	92	84
query62	614	374	352	352
query63	286	257	273	257
query64	5293	1774	1769	1769
query65	3332	3258	3249	3249
query66	1284	323	317	317
query67	15563	15071	15103	15071
query68	12336	545	514	514
query69	606	302	305	302
query70	1648	1418	1468	1418
query71	512	205	215	205
query72	4869	2876	2854	2854
query73	2724	318	313	313
query74	6962	6366	6440	6366
query75	5125	2313	2343	2313
query76	6641	1078	1033	1033
query77	657	234	234	234
query78	9013	8827	8646	8646
query79	1012	498	505	498
query80	565	320	312	312
query81	446	218	201	201
query82	205	84	78	78
query83	139	129	119	119
query84	276	69	69	69
query85	1075	335	325	325
query86	386	379	368	368
query87	3492	3381	3355	3355
query88	2984	2187	2186	2186
query89	460	348	355	348
query90	2001	190	186	186
query91	164	124	138	124
query92	56	46	44	44
query93	1721	460	423	423
query94	1221	158	158	158
query95	489	451	448	448
query96	620	316	321	316
query97	4222	4176	4162	4162
query98	205	185	184	184
query99	1023	715	703	703
Total cold run time: 296180 ms
Total hot run time: 176724 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.74 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c1a745e978e3d6730638d60df4fc275641b7944b, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.02	0.02
query3	0.23	0.05	0.05
query4	1.70	0.07	0.08
query5	0.54	0.51	0.52
query6	1.30	0.64	0.63
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.54	0.51	0.49
query10	0.55	0.55	0.56
query11	0.11	0.09	0.08
query12	0.11	0.09	0.08
query13	0.61	0.61	0.60
query14	0.78	0.82	0.80
query15	0.80	0.77	0.79
query16	0.36	0.36	0.39
query17	1.04	1.00	1.01
query18	0.22	0.27	0.22
query19	1.78	1.71	1.69
query20	0.01	0.01	0.01
query21	15.42	0.56	0.59
query22	2.29	2.56	1.50
query23	17.26	0.83	0.86
query24	2.40	0.27	1.13
query25	0.38	0.12	0.13
query26	0.41	0.15	0.14
query27	0.07	0.06	0.05
query28	12.18	0.77	0.77
query29	12.56	3.24	3.17
query30	0.52	0.46	0.49
query31	2.78	0.34	0.34
query32	3.39	0.48	0.48
query33	3.21	3.22	3.27
query34	16.12	4.38	4.24
query35	4.34	4.36	4.28
query36	1.12	1.07	1.07
query37	0.06	0.06	0.06
query38	0.04	0.03	0.03
query39	0.02	0.02	0.01
query40	0.16	0.14	0.13
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.03	0.01	0.02
Total cold run time: 105.68 s
Total hot run time: 29.74 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c1a745e978e3d6730638d60df4fc275641b7944b with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       15.0 seconds inserted 10000000 Rows, about 666K ops/s

@morrySnow morrySnow merged commit 25da056 into apache:master Jan 22, 2024
morningman pushed a commit that referenced this pull request Jan 22, 2024
Before, drop stats operation need to call columns * followers times of isMaster() function and the same times of rpc to drop remote column stats. This pr is to reduce the rpc calls and use more efficient way to check master node instead of using isMaster()

backport #30144
seawinde pushed a commit to seawinde/doris that referenced this pull request Jan 22, 2024
Before, drop stats operation need to call columns * followers times of isMaster() function and the same times of rpc to drop remote column stats. This pr is to reduce the rpc calls and use more efficient way to check master node instead of using isMaster()
yiguolei pushed a commit that referenced this pull request Jan 23, 2024
Before, drop stats operation need to call columns * followers times of isMaster() function and the same times of rpc to drop remote column stats. This pr is to reduce the rpc calls and use more efficient way to check master node instead of using isMaster()
@Jibing-Li Jibing-Li deleted the dropstats branch January 24, 2024 14:23
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Before, drop stats operation need to call columns * followers times of isMaster() function and the same times of rpc to drop remote column stats. This pr is to reduce the rpc calls and use more efficient way to check master node instead of using isMaster()

backport apache#30144
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.5-merged dev/3.0.0-merged p0_c reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants

Comments