Skip to content

[fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0.#30100

Merged
xiaokang merged 1 commit intoapache:branch-2.0from
Jibing-Li:health2.0
Jan 19, 2024
Merged

[fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0.#30100
xiaokang merged 1 commit intoapache:branch-2.0from
Jibing-Li:health2.0

Conversation

@Jibing-Li
Copy link
Contributor

backport #30096

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@Jibing-Li Jibing-Li marked this pull request as ready for review January 18, 2024 07:15
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50816 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5b2acd7ac1d9e3d00e378bbaeabca3b40a7455a1, data reload: false

------ Round 1 ----------------------------------
q1	17780	4632	4561	4561
q2	2031	146	141	141
q3	10459	1941	1967	1941
q4	10353	1280	1379	1280
q5	8610	3957	4026	3957
q6	262	123	122	122
q7	2094	1578	1626	1578
q8	9379	2842	2892	2842
q9	11023	10706	10636	10636
q10	8747	3599	3586	3586
q11	421	248	241	241
q12	492	305	298	298
q13	18314	3954	4028	3954
q14	359	318	321	318
q15	513	455	450	450
q16	713	591	599	591
q17	1182	1012	1000	1000
q18	7177	6989	6798	6798
q19	1749	1653	1590	1590
q20	543	304	317	304
q21	4636	4271	4245	4245
q22	509	414	383	383
Total cold run time: 117346 ms
Total hot run time: 50816 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4594	4528	4519	4519
q2	337	225	223	223
q3	4238	4211	4220	4211
q4	2866	2845	2843	2843
q5	7069	7020	7054	7020
q6	251	119	118	118
q7	3262	2838	2839	2838
q8	4257	4352	4379	4352
q9	17268	17066	17100	17066
q10	4262	4325	4319	4319
q11	823	664	662	662
q12	1047	842	852	842
q13	7254	3763	3790	3763
q14	453	423	423	423
q15	511	456	455	455
q16	756	714	705	705
q17	3835	3911	3872	3872
q18	8959	8866	8978	8866
q19	1816	1814	1809	1809
q20	2429	2171	2076	2076
q21	8644	8619	8590	8590
q22	1026	945	935	935
Total cold run time: 85957 ms
Total hot run time: 80507 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 238478 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5b2acd7ac1d9e3d00e378bbaeabca3b40a7455a1, data reload: false

query1	918	384	379	379
query2	6542	2183	2123	2123
query3	6929	201	205	201
query4	20864	17935	17872	17872
query5	19711	6269	6234	6234
query6	285	223	224	223
query7	4159	291	307	291
query8	242	239	239	239
query9	3149	2695	2645	2645
query10	412	301	304	301
query11	11275	10632	10679	10632
query12	124	73	69	69
query13	5662	659	611	611
query14	18010	13483	13319	13319
query15	380	239	238	238
query16	6445	273	267	267
query17	3206	870	847	847
query18	2267	404	406	404
query19	205	144	148	144
query20	77	78	77	77
query21	180	92	94	92
query22	5347	5033	5059	5033
query23	32488	31997	32092	31997
query24	6960	6468	6461	6461
query25	539	428	418	418
query26	527	161	154	154
query27	1854	304	291	291
query28	6078	2281	2241	2241
query29	2808	2841	2799	2799
query30	231	164	159	159
query31	893	738	743	738
query32	62	55	58	55
query33	390	243	241	241
query34	844	459	485	459
query35	1109	925	887	887
query36	1491	1414	1443	1414
query37	85	62	59	59
query38	3106	2961	2952	2952
query39	1370	1335	1302	1302
query40	208	95	98	95
query41	34	31	31	31
query42	87	82	85	82
query43	617	540	652	540
query44	1136	714	716	714
query45	242	230	227	227
query46	1237	974	943	943
query47	1977	1657	1743	1657
query48	973	697	670	670
query49	615	370	354	354
query50	870	575	610	575
query51	5649	5542	5494	5494
query52	84	91	83	83
query53	445	320	317	317
query54	2432	2279	2253	2253
query55	81	79	79	79
query56	204	181	184	181
query57	1167	1091	1080	1080
query58	212	197	210	197
query59	3380	3132	3117	3117
query60	206	192	189	189
query61	83	82	86	82
query62	897	481	472	472
query63	485	337	339	337
query64	1935	1283	1284	1283
query65	3735	3654	3597	3597
query66	831	378	361	361
query67	16625	15412	15328	15328
query68	9130	657	681	657
query69	571	342	355	342
query70	1906	1666	1951	1666
query71	418	316	318	316
query72	4399	3379	3357	3357
query73	725	320	322	320
query74	6364	5853	5858	5853
query75	4915	3582	3488	3488
query76	5586	1191	1187	1187
query77	886	261	250	250
query78	31162	52888	48036	48036
query79	14972	638	653	638
query80	4398	394	378	378
query81	572	232	227	227
query82	606	101	95	95
query83	403	140	129	129
query84	255	67	71	67
query85	2377	278	273	273
query86	443	387	371	371
query87	3264	2988	2989	2988
query88	6452	2328	2312	2312
query89	383	292	309	292
query90	2502	199	210	199
query91	163	119	117	117
query92	69	51	51	51
query93	4498	592	580	580
query94	1702	201	205	201
query95	1129	1072	1078	1072
query96	644	341	327	327
query97	6536	6363	6386	6363
query98	187	175	181	175
query99	3862	945	1087	945
Total cold run time: 345150 ms
Total hot run time: 238478 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.14 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5b2acd7ac1d9e3d00e378bbaeabca3b40a7455a1, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.05
query4	1.80	0.06	0.07
query5	0.53	0.53	0.52
query6	1.24	0.62	0.62
query7	0.01	0.01	0.01
query8	0.04	0.03	0.03
query9	0.52	0.48	0.48
query10	0.54	0.54	0.53
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.62	0.63	0.62
query14	0.78	0.78	0.80
query15	0.79	0.76	0.78
query16	0.36	0.38	0.35
query17	0.99	1.01	1.01
query18	0.23	0.26	0.24
query19	1.85	1.81	1.84
query20	0.02	0.01	0.00
query21	15.48	0.58	0.56
query22	1.95	2.24	1.29
query23	16.88	0.96	0.94
query24	6.03	2.82	2.56
query25	2.38	0.15	0.15
query26	0.14	0.14	0.15
query27	0.14	0.16	0.15
query28	4.75	0.75	0.78
query29	12.72	2.34	2.27
query30	0.57	0.55	0.48
query31	2.80	0.39	0.37
query32	3.39	0.48	0.51
query33	3.08	3.09	3.06
query34	15.21	4.79	4.80
query35	4.86	4.87	4.81
query36	1.07	1.02	1.01
query37	0.06	0.05	0.04
query38	0.05	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 102.73 s
Total hot run time: 32.14 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 5b2acd7ac1d9e3d00e378bbaeabca3b40a7455a1 with default session variables
Stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@Jibing-Li Jibing-Li changed the title Reanalyze olapTable if getRowCount is not 0 and last time row count is 0. [fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0. Jan 19, 2024
@xiaokang xiaokang merged commit 7ab22bd into apache:branch-2.0 Jan 19, 2024
@Jibing-Li Jibing-Li deleted the health2.0 branch January 19, 2024 11:46
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants