Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Opt] multi table func exec performance #34090

Merged
merged 2 commits into from
Apr 29, 2024
Merged

[Opt] multi table func exec performance #34090

merged 2 commits into from
Apr 29, 2024

Conversation

HappenLee
Copy link
Contributor

@HappenLee HappenLee commented Apr 24, 2024 •
edited

Proposed changes

before：

[regression_test_query_p0_sql_functions_table_function]>select count(*) from test_lv_str                         lateral view explode_numbers(10000) tmp1 as e1                         lateral view explode_numbers(10000) tmp2 as e2;
+-----------+
| count(*)  |
+-----------+
| 100000000 |
+-----------+
1 row in set (2.32 sec)

after:

[regression_test_query_p0_sql_functions_table_function]>select count(*) from test_lv_str                         lateral view explode_numbers(10000) tmp1 as e1                         lateral view explode_numbers(10000) tmp2 as e2;
+-----------+
| count(*)  |
+-----------+
| 100000000 |
+-----------+
1 row in set (0.36 sec)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@HappenLee
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@HappenLee
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@HappenLee
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.20% (8919/25341)
Line Coverage: 26.97% (73325/271886)
Region Coverage: 26.15% (37879/144873)
Branch Coverage: 22.96% (19286/83986)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2204027b91808cf3d59d8725fa09c49eb80d3b37_2204027b91808cf3d59d8725fa09c49eb80d3b37/report/index.html

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@HappenLee
Copy link
Contributor Author

run buildall

BiteTheDDDDt
BiteTheDDDDt reviewed Apr 25, 2024
View reviewed changes
regression-test/conf/regression-conf.groovy Outdated
@@ -24,8 +24,8 @@ defaultDb = "regression_test"
// init cmd like: select @@session.tx_read_only
// at each time we connect.
// add allowLoadLocalInfile so that the jdbc can execute mysql load data from client.
jdbcUrl = "jdbc:mysql://127.0.0.1:9030/?useLocalSessionState=true&allowLoadLocalInfile=true"
targetJdbcUrl = "jdbc:mysql://127.0.0.1:9030/?useLocalSessionState=true&allowLoadLocalInfile=true"
jdbcUrl = "jdbc:mysql://127.0.0.1:9137/?useLocalSessionState=true&allowLoadLocalInfile=true"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need reset

@HappenLee
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@HappenLee
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.53% (8929/25134)
Line Coverage: 27.14% (73455/270651)
Region Coverage: 26.33% (37947/144115)
Branch Coverage: 23.13% (19333/83598)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2b7fc023143f50be3828d34138b9273c932f3c81_2b7fc023143f50be3828d34138b9273c932f3c81/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 188802 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b7fc023143f50be3828d34138b9273c932f3c81, data reload: false

query1	906	362	350	350
query2	6187	2420	2403	2403
query3	6648	205	212	205
query4	22870	22012	21901	21901
query5	3853	436	429	429
query6	278	191	191	191
query7	4538	294	296	294
query8	243	191	188	188
query9	8438	2370	2357	2357
query10	406	241	281	241
query11	15527	14880	14780	14780
query12	122	88	88	88
query13	1613	355	370	355
query14	9224	8586	8616	8586
query15	285	174	179	174
query16	8449	260	263	260
query17	1861	570	546	546
query18	2090	271	266	266
query19	321	143	160	143
query20	94	81	79	79
query21	192	124	126	124
query22	4990	4808	4807	4807
query23	33738	33405	33278	33278
query24	10878	2896	3009	2896
query25	594	378	363	363
query26	1176	144	143	143
query27	2688	310	309	309
query28	7493	2005	1993	1993
query29	870	597	577	577
query30	247	150	152	150
query31	965	728	737	728
query32	94	51	53	51
query33	745	254	251	251
query34	1000	475	472	472
query35	822	690	667	667
query36	1079	944	948	944
query37	140	80	68	68
query38	3186	3024	2990	2990
query39	1595	1553	1644	1553
query40	195	131	132	131
query41	43	38	38	38
query42	106	98	96	96
query43	558	561	559	559
query44	1205	738	735	735
query45	284	273	269	269
query46	1077	724	707	707
query47	1937	1847	1853	1847
query48	379	303	299	299
query49	832	408	399	399
query50	767	391	394	391
query51	6847	6596	6582	6582
query52	109	92	93	92
query53	350	278	276	276
query54	302	249	241	241
query55	78	72	72	72
query56	248	234	228	228
query57	1202	1162	1146	1146
query58	238	211	200	200
query59	3577	3362	3184	3184
query60	270	243	249	243
query61	112	107	110	107
query62	635	440	432	432
query63	307	287	286	286
query64	8586	7217	7245	7217
query65	3083	3145	3036	3036
query66	947	336	335	335
query67	15846	14808	15004	14808
query68	10663	532	551	532
query69	631	301	299	299
query70	1381	1151	1110	1110
query71	532	273	268	268
query72	9290	2631	2456	2456
query73	1763	319	318	318
query74	6615	6152	6171	6152
query75	6139	2698	2638	2638
query76	6700	1000	1077	1000
query77	713	264	264	264
query78	11033	10496	10226	10226
query79	11751	510	519	510
query80	1767	434	428	428
query81	492	232	221	221
query82	181	92	88	88
query83	209	168	159	159
query84	260	93	92	92
query85	1137	316	257	257
query86	340	293	298	293
query87	3386	3114	3086	3086
query88	5973	2332	2342	2332
query89	501	367	371	367
query90	2345	176	177	176
query91	121	96	98	96
query92	58	46	46	46
query93	6175	498	500	498
query94	1601	175	178	175
query95	397	306	288	288
query96	616	269	263	263
query97	3131	2923	2920	2920
query98	229	212	215	212
query99	1062	889	876	876
Total cold run time: 313757 ms
Total hot run time: 188802 ms

@HappenLee
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.53% (8929/25134)
Line Coverage: 27.15% (73475/270660)
Region Coverage: 26.34% (37963/144125)
Branch Coverage: 23.13% (19337/83606)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e13775c6a0612e93baa38c895325f4cb59e85384_e13775c6a0612e93baa38c895325f4cb59e85384/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41203 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e13775c6a0612e93baa38c895325f4cb59e85384, data reload: false

------ Round 1 ----------------------------------
q1	17616	4294	4238	4238
q2	2011	203	192	192
q3	10488	1163	1216	1163
q4	10204	817	762	762
q5	7517	2772	2746	2746
q6	217	137	129	129
q7	1073	617	637	617
q8	9225	2113	2093	2093
q9	9354	6755	6787	6755
q10	9019	3963	3882	3882
q11	466	242	240	240
q12	469	222	232	222
q13	17377	3198	3123	3123
q14	284	233	248	233
q15	520	473	476	473
q16	506	423	400	400
q17	993	740	782	740
q18	8274	7759	7722	7722
q19	6761	1532	1565	1532
q20	655	320	324	320
q21	5185	3349	4152	3349
q22	357	293	272	272
Total cold run time: 118571 ms
Total hot run time: 41203 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4594	4304	4399	4304
q2	389	294	280	280
q3	3154	2914	2919	2914
q4	1919	1658	1680	1658
q5	5456	5416	5551	5416
q6	214	121	123	121
q7	2353	2001	2002	2001
q8	3223	3423	3455	3423
q9	8804	8807	8956	8807
q10	4100	3912	3742	3742
q11	576	480	482	480
q12	763	615	600	600
q13	16588	3084	3103	3084
q14	320	276	299	276
q15	528	485	496	485
q16	504	428	435	428
q17	1786	1469	1550	1469
q18	8053	7502	7356	7356
q19	3038	1514	1571	1514
q20	2027	1751	1747	1747
q21	13343	4982	4716	4716
q22	551	485	479	479
Total cold run time: 82283 ms
Total hot run time: 55300 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186926 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e13775c6a0612e93baa38c895325f4cb59e85384, data reload: false

query1	909	359	363	359
query2	6455	2411	2382	2382
query3	6643	205	216	205
query4	24375	21140	21188	21140
query5	4147	408	420	408
query6	265	173	172	172
query7	4593	298	287	287
query8	253	189	185	185
query9	8426	2327	2335	2327
query10	445	240	250	240
query11	14905	14205	14170	14170
query12	140	88	86	86
query13	1644	364	373	364
query14	10535	8156	8129	8129
query15	236	168	175	168
query16	7974	257	258	257
query17	1787	555	541	541
query18	2043	286	264	264
query19	203	150	155	150
query20	96	93	89	89
query21	198	135	127	127
query22	5018	4860	4846	4846
query23	33764	33024	33180	33024
query24	7463	2941	3022	2941
query25	588	367	364	364
query26	701	151	148	148
query27	2203	313	310	310
query28	5199	2014	2021	2014
query29	839	610	596	596
query30	242	151	153	151
query31	966	745	726	726
query32	82	51	52	51
query33	553	256	245	245
query34	887	474	477	474
query35	775	680	684	680
query36	1023	906	903	903
query37	102	68	65	65
query38	3175	2990	3045	2990
query39	1589	1557	1528	1528
query40	204	127	124	124
query41	43	38	39	38
query42	103	96	101	96
query43	593	539	538	538
query44	1077	735	751	735
query45	280	257	263	257
query46	1083	724	722	722
query47	1917	1845	1838	1838
query48	367	302	290	290
query49	877	409	397	397
query50	774	381	392	381
query51	6909	6711	6762	6711
query52	103	86	90	86
query53	349	282	273	273
query54	262	225	235	225
query55	78	68	72	68
query56	244	220	220	220
query57	1173	1141	1140	1140
query58	219	216	201	201
query59	3439	3392	3272	3272
query60	253	232	231	231
query61	90	90	88	88
query62	590	447	437	437
query63	306	275	278	275
query64	8231	7174	7131	7131
query65	3094	3038	3063	3038
query66	774	331	335	331
query67	15160	14965	15043	14965
query68	5208	543	537	537
query69	477	299	297	297
query70	1196	1098	1161	1098
query71	366	265	270	265
query72	7846	2739	2545	2545
query73	699	325	329	325
query74	6573	6103	6067	6067
query75	3384	2695	2663	2663
query76	2745	1025	997	997
query77	394	270	270	270
query78	10782	10223	10186	10186
query79	4672	527	519	519
query80	2199	442	447	442
query81	523	230	224	224
query82	1141	96	97	96
query83	302	172	174	172
query84	267	92	92	92
query85	1893	325	309	309
query86	480	277	284	277
query87	3337	3051	3148	3051
query88	4983	2341	2336	2336
query89	498	375	374	374
query90	2059	188	283	188
query91	125	95	96	95
query92	60	48	46	46
query93	5405	511	507	507
query94	1306	179	175	175
query95	400	304	291	291
query96	598	261	261	261
query97	3161	2958	2975	2958
query98	240	216	217	216
query99	1205	868	833	833
Total cold run time: 284557 ms
Total hot run time: 186926 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@HappenLee HappenLee merged commit 167e85e into apache:master Apr 29, 2024
24 of 28 checks passed
M1saka2003 pushed a commit to M1saka2003/doris that referenced this pull request May 14, 2024
@HappenLee @M1saka2003
[Opt] multi table func exec performance (apache#34090)
ee916f4
M1saka2003 pushed a commit to M1saka2003/doris that referenced this pull request May 24, 2024
@HappenLee @M1saka2003
[Opt] multi table func exec performance (apache#34090)
e22d8af
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt left review comments

@zclllyybb zclllyybb zclllyybb approved these changes

@zhangstar333 zhangstar333 zhangstar333 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@HappenLee @doris-robot @BiteTheDDDDt @zclllyybb @zhangstar333