Skip to content

[fix](statistics)Disable fetch min/max column stats through HMS, because the value may inaccurate and misleading. (#35124)#35293

Merged
Jibing-Li merged 1 commit intoapache:branch-2.0from
Jibing-Li:minmax2.0
May 23, 2024
Merged

[fix](statistics)Disable fetch min/max column stats through HMS, because the value may inaccurate and misleading. (#35124)#35293
Jibing-Li merged 1 commit intoapache:branch-2.0from
Jibing-Li:minmax2.0

Conversation

@Jibing-Li
Copy link
Copy Markdown
Contributor

backport #35124

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li Jibing-Li marked this pull request as ready for review May 23, 2024 09:48
@Jibing-Li
Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 49537 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66d89eee0a7eb6ef56580189d286e49a19605ee9, data reload: false

------ Round 1 ----------------------------------
q1	17562	4372	4383	4372
q2	2029	153	143	143
q3	10280	1912	1896	1896
q4	10360	1224	1312	1224
q5	8570	3862	3893	3862
q6	238	126	127	126
q7	2026	1620	1596	1596
q8	9498	2707	2688	2688
q9	13879	11079	10141	10141
q10	8628	3514	3473	3473
q11	413	246	251	246
q12	469	302	301	301
q13	18331	3977	4047	3977
q14	345	335	324	324
q15	509	453	460	453
q16	674	577	573	573
q17	1139	955	904	904
q18	7287	7043	6915	6915
q19	1676	1599	1547	1547
q20	533	323	304	304
q21	4430	4146	4085	4085
q22	499	393	387	387
Total cold run time: 119375 ms
Total hot run time: 49537 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4290	4366	4253	4253
q2	319	226	218	218
q3	4178	4155	4152	4152
q4	2726	2752	2741	2741
q5	7172	7087	7165	7087
q6	238	122	120	120
q7	3225	2829	2830	2829
q8	4329	4453	4465	4453
q9	16945	16721	16785	16721
q10	4204	4232	4237	4232
q11	748	669	712	669
q12	1038	869	842	842
q13	6934	3726	3759	3726
q14	456	434	425	425
q15	500	455	465	455
q16	740	689	688	688
q17	3851	3808	3814	3808
q18	8801	8745	8847	8745
q19	1715	1716	1691	1691
q20	2411	2139	2122	2122
q21	8466	8552	8658	8552
q22	1002	960	926	926
Total cold run time: 84288 ms
Total hot run time: 79455 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 203370 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 66d89eee0a7eb6ef56580189d286e49a19605ee9, data reload: false

query1	917	386	376	376
query2	6528	2722	2852	2722
query3	6922	211	209	209
query4	20060	17967	17954	17954
query5	19736	6572	6493	6493
query6	282	226	226	226
query7	4153	313	317	313
query8	261	279	243	243
query9	3136	2714	2617	2617
query10	420	309	313	309
query11	11334	10667	10810	10667
query12	122	76	72	72
query13	5586	701	697	697
query14	17809	13109	13494	13109
query15	378	235	229	229
query16	6473	270	256	256
query17	1724	1473	863	863
query18	2335	402	401	401
query19	219	149	155	149
query20	78	77	81	77
query21	188	97	93	93
query22	5176	5057	5033	5033
query23	32530	31840	31828	31828
query24	6954	6665	6503	6503
query25	521	423	425	423
query26	531	164	162	162
query27	1874	297	293	293
query28	6139	2373	2323	2323
query29	3021	2929	2780	2780
query30	238	165	167	165
query31	901	766	738	738
query32	64	66	63	63
query33	403	249	247	247
query34	852	490	477	477
query35	1139	929	931	929
query36	1244	1202	1197	1197
query37	90	60	58	58
query38	3091	2907	2960	2907
query39	1362	1348	1307	1307
query40	206	95	96	95
query41	39	37	36	36
query42	82	81	83	81
query43	737	764	713	713
query44	1173	709	721	709
query45	248	232	227	227
query46	1222	961	986	961
query47	1842	1770	1704	1704
query48	1005	725	712	712
query49	617	381	366	366
query50	874	622	646	622
query51	4739	4706	4658	4658
query52	95	96	76	76
query53	452	316	314	314
query54	2659	2484	2464	2464
query55	89	89	80	80
query56	243	212	193	193
query57	1236	1107	1138	1107
query58	220	204	199	199
query59	4303	3785	3902	3785
query60	216	193	201	193
query61	87	84	86	84
query62	838	484	433	433
query63	481	343	338	338
query64	2537	1484	1286	1286
query65	3637	3563	3590	3563
query66	810	395	371	371
query67	16262	15624	16058	15624
query68	10366	677	656	656
query69	569	332	345	332
query70	2034	1545	1404	1404
query71	411	321	329	321
query72	6538	3396	3437	3396
query73	729	327	328	327
query74	6324	5845	5805	5805
query75	5338	3667	3654	3654
query76	6670	1146	1217	1146
query77	1169	253	254	253
query78	12474	12261	11815	11815
query79	10964	660	651	651
query80	701	396	407	396
query81	487	235	229	229
query82	661	97	98	97
query83	190	141	129	129
query84	260	71	69	69
query85	785	304	304	304
query86	345	294	309	294
query87	3225	3012	3062	3012
query88	5109	2352	2360	2352
query89	386	305	284	284
query90	2036	209	201	201
query91	165	133	158	133
query92	61	52	55	52
query93	4484	586	553	553
query94	745	208	208	208
query95	1144	1065	1049	1049
query96	651	338	336	336
query97	6513	6462	6372	6372
query98	195	178	170	170
query99	2906	874	868	868
Total cold run time: 316375 ms
Total hot run time: 203370 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 30.87 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 66d89eee0a7eb6ef56580189d286e49a19605ee9, data reload: false

query1	0.02	0.02	0.01
query2	0.07	0.03	0.02
query3	0.25	0.04	0.04
query4	1.82	0.07	0.06
query5	0.53	0.53	0.52
query6	1.27	0.66	0.62
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.48	0.48
query10	0.53	0.53	0.55
query11	0.12	0.08	0.09
query12	0.11	0.09	0.09
query13	0.61	0.62	0.61
query14	0.79	0.78	0.79
query15	0.78	0.76	0.77
query16	0.35	0.38	0.38
query17	1.00	1.04	1.03
query18	0.24	0.26	0.23
query19	1.94	1.83	1.84
query20	0.02	0.01	0.01
query21	15.47	0.56	0.55
query22	2.19	1.83	1.50
query23	16.89	0.96	0.96
query24	7.33	1.27	1.21
query25	0.39	0.07	0.06
query26	0.80	0.15	0.16
query27	0.05	0.04	0.04
query28	5.40	0.73	0.74
query29	12.65	2.35	2.27
query30	0.59	0.50	0.51
query31	2.81	0.39	0.37
query32	3.36	0.49	0.49
query33	3.06	3.06	3.07
query34	15.27	4.81	4.78
query35	4.86	4.84	4.85
query36	1.04	1.02	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.15	0.14	0.13
query41	0.07	0.01	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 103.56 s
Total hot run time: 30.87 s

@doris-robot
Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 66d89eee0a7eb6ef56580189d286e49a19605ee9 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@Jibing-Li Jibing-Li merged commit cd689b6 into apache:branch-2.0 May 23, 2024
@Jibing-Li Jibing-Li deleted the minmax2.0 branch May 28, 2024 08:34
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants