Skip to content

[improvement](iceberg/paimon)support estimate row count for 2.0#31927

Merged
morningman merged 1 commit intoapache:branch-2.0from
wuwenchi:row-count-2.0
Mar 7, 2024
Merged

[improvement](iceberg/paimon)support estimate row count for 2.0#31927
morningman merged 1 commit intoapache:branch-2.0from
wuwenchi:row-count-2.0

Conversation

@wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented Mar 7, 2024

Proposed changes

mirror: #31204 #31473

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@github-actions github-actions bot added area/planner Issues or PRs related to the query planner kind/test labels Mar 7, 2024
@wuwenchi
Copy link
Contributor Author

wuwenchi commented Mar 7, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49920 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5b9c0283ad37d61fb6ac41d50a775b537c734374, data reload: false

------ Round 1 ----------------------------------
q1	17740	4407	4373	4373
q2	2029	148	139	139
q3	10329	1875	1905	1875
q4	10359	1247	1326	1247
q5	8383	3925	3960	3925
q6	225	124	120	120
q7	2022	1604	1609	1604
q8	9523	2688	2688	2688
q9	13217	10867	10698	10698
q10	8644	3530	3480	3480
q11	417	234	244	234
q12	457	290	301	290
q13	18315	3954	4029	3954
q14	352	328	334	328
q15	504	457	455	455
q16	696	586	588	586
q17	1107	939	902	902
q18	7178	6705	6897	6705
q19	1703	1566	1586	1566
q20	505	298	298	298
q21	4383	4046	4060	4046
q22	509	407	409	407
Total cold run time: 118597 ms
Total hot run time: 49920 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4330	4307	4244	4244
q2	311	216	214	214
q3	4142	4149	4106	4106
q4	2786	2734	2751	2734
q5	7303	7199	7214	7199
q6	234	118	115	115
q7	3255	2873	2824	2824
q8	4324	4413	4467	4413
q9	17739	17233	17244	17233
q10	4248	4268	4280	4268
q11	743	707	684	684
q12	1037	860	844	844
q13	6756	3765	3734	3734
q14	450	419	430	419
q15	498	464	455	455
q16	757	702	717	702
q17	3788	3907	3906	3906
q18	9271	9268	8930	8930
q19	1723	1653	1687	1653
q20	2394	2203	2178	2178
q21	8403	8455	8569	8455
q22	1041	982	926	926
Total cold run time: 85533 ms
Total hot run time: 80236 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 201628 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5b9c0283ad37d61fb6ac41d50a775b537c734374, data reload: false

query1	927	388	381	381
query2	6518	2198	2164	2164
query3	6919	200	198	198
query4	20798	17939	17781	17781
query5	19731	6492	6504	6492
query6	317	214	234	214
query7	4381	295	287	287
query8	249	216	232	216
query9	3104	2689	2617	2617
query10	422	288	303	288
query11	11292	10618	10710	10618
query12	125	73	69	69
query13	5589	651	634	634
query14	18361	13424	13476	13424
query15	360	226	238	226
query16	6444	268	268	268
query17	1572	1445	872	872
query18	2308	404	395	395
query19	197	140	147	140
query20	77	77	75	75
query21	183	92	97	92
query22	5153	5139	5109	5109
query23	32401	31990	31810	31810
query24	6897	6494	6525	6494
query25	508	414	404	404
query26	511	163	157	157
query27	1786	287	291	287
query28	6089	2246	2202	2202
query29	2786	2677	2647	2647
query30	234	157	161	157
query31	910	707	735	707
query32	64	60	56	56
query33	391	251	248	248
query34	852	455	465	455
query35	1105	892	880	880
query36	1733	1414	1564	1414
query37	90	62	60	60
query38	3049	2873	2937	2873
query39	1393	1302	1304	1302
query40	204	84	89	84
query41	36	32	31	31
query42	91	84	88	84
query43	653	645	594	594
query44	1116	726	715	715
query45	240	226	226	226
query46	1228	977	950	950
query47	1811	1719	1881	1719
query48	975	695	662	662
query49	613	369	364	364
query50	853	596	601	596
query51	4712	4682	4648	4648
query52	80	76	72	72
query53	446	310	316	310
query54	2645	2460	2527	2460
query55	87	68	78	68
query56	201	184	191	184
query57	1174	1192	1104	1104
query58	212	198	199	198
query59	3743	3192	3427	3192
query60	210	197	186	186
query61	87	84	88	84
query62	798	536	473	473
query63	492	337	330	330
query64	2234	1495	1461	1461
query65	3591	3576	3566	3566
query66	763	374	364	364
query67	17109	15403	16157	15403
query68	8223	649	656	649
query69	556	353	345	345
query70	2006	1784	1512	1512
query71	409	305	309	305
query72	6444	3444	3412	3412
query73	722	321	324	321
query74	6258	5814	5881	5814
query75	4677	3664	3717	3664
query76	4617	1146	1202	1146
query77	536	249	255	249
query78	13020	11685	12691	11685
query79	9690	632	627	627
query80	2532	392	387	387
query81	534	229	236	229
query82	1422	95	92	92
query83	178	129	135	129
query84	255	68	67	67
query85	1359	286	280	280
query86	405	389	416	389
query87	3252	3007	3014	3007
query88	5008	2291	2296	2291
query89	387	308	271	271
query90	1757	195	195	195
query91	149	115	115	115
query92	55	49	52	49
query93	5417	580	558	558
query94	908	202	193	193
query95	1092	1051	1062	1051
query96	624	317	308	308
query97	6427	6375	6445	6375
query98	189	178	190	178
query99	2715	905	894	894
Total cold run time: 314053 ms
Total hot run time: 201628 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.74 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5b9c0283ad37d61fb6ac41d50a775b537c734374, data reload: false

query1	0.03	0.02	0.02
query2	0.06	0.02	0.02
query3	0.25	0.05	0.04
query4	1.83	0.07	0.05
query5	0.53	0.52	0.51
query6	1.25	0.62	0.62
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.53	0.48	0.48
query10	0.54	0.55	0.54
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.62	0.62	0.62
query14	0.78	0.79	0.79
query15	0.77	0.75	0.77
query16	0.37	0.35	0.37
query17	1.02	1.01	1.02
query18	0.22	0.24	0.25
query19	1.90	1.82	1.77
query20	0.02	0.01	0.01
query21	15.48	0.56	0.56
query22	2.29	3.14	0.91
query23	17.24	0.93	0.82
query24	4.78	1.71	0.73
query25	0.35	0.13	0.06
query26	0.58	0.15	0.16
query27	0.03	0.04	0.04
query28	8.46	0.75	0.70
query29	12.76	2.35	2.39
query30	0.62	0.54	0.54
query31	2.81	0.39	0.37
query32	3.41	0.51	0.50
query33	3.03	3.07	3.09
query34	15.30	4.81	4.80
query35	4.86	4.84	4.86
query36	1.07	1.01	1.03
query37	0.06	0.04	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.15	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 104.43 s
Total hot run time: 29.74 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 5b9c0283ad37d61fb6ac41d50a775b537c734374 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.7 seconds inserted 10000000 Rows, about 483K ops/s

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 5d4c030 into apache:branch-2.0 Mar 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/planner Issues or PRs related to the query planner kind/test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants