Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](expr) fix performance problem caused by too many virtual function call #28508

Merged
merged 3 commits into from
Dec 18, 2023
Merged

[fix](expr) fix performance problem caused by too many virtual function call #28508

merged 3 commits into from
Dec 18, 2023

Conversation

jacktengg
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@jacktengg
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit bb7e7405b28f4e9fff95dd8a8a195732f9a0f4f6, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4772	4428	4470	4428
q2	365	171	148	148
q3	1474	1269	1215	1215
q4	1111	901	897	897
q5	3154	3152	3147	3147
q6	245	127	129	127
q7	993	491	491	491
q8	2210	2210	2175	2175
q9	6683	6690	6668	6668
q10	3213	3274	3293	3274
q11	326	213	198	198
q12	356	213	210	210
q13	4532	3789	3799	3789
q14	243	213	207	207
q15	580	524	528	524
q16	444	391	386	386
q17	1031	614	539	539
q18	7044	6768	6865	6768
q19	1527	1409	1426	1409
q20	548	314	282	282
q21	3085	2672	2600	2600
q22	349	287	285	285
Total cold run time: 44285 ms
Total hot run time: 39767 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4402	4396	4383	4383
q2	265	164	172	164
q3	3493	3505	3481	3481
q4	2386	2370	2363	2363
q5	5668	5676	5659	5659
q6	240	123	121	121
q7	2376	1839	1833	1833
q8	3530	3517	3532	3517
q9	9042	8938	8984	8938
q10	3899	3977	4015	3977
q11	505	373	385	373
q12	768	597	587	587
q13	4291	3603	3588	3588
q14	284	258	259	258
q15	578	521	520	520
q16	521	455	460	455
q17	1896	1848	1848	1848
q18	8410	8121	8163	8121
q19	1765	1776	1746	1746
q20	2249	1969	1937	1937
q21	6467	6166	6142	6142
q22	523	433	427	427
Total cold run time: 63558 ms
Total hot run time: 60438 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.91 seconds
stream load tsv: 584 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17219497364 Bytes

@jacktengg jacktengg force-pushed the fix1216 branch 3 times, most recently from 19e92e7 to b1d1d53 Compare December 16, 2023 12:07
@jacktengg
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit b1d1d53c5c350455daff001a316481b357da452b, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4704	4408	4468	4408
q2	359	144	139	139
q3	1441	1241	1199	1199
q4	1105	869	858	858
q5	3119	3122	3133	3122
q6	244	127	128	127
q7	993	477	481	477
q8	2177	2197	2160	2160
q9	6671	6636	6690	6636
q10	3234	3293	3240	3240
q11	323	209	207	207
q12	353	204	209	204
q13	4590	3780	3848	3780
q14	240	213	210	210
q15	570	530	519	519
q16	431	377	382	377
q17	1009	639	571	571
q18	7211	6948	6950	6948
q19	1536	1443	1390	1390
q20	534	294	307	294
q21	3044	2621	2662	2621
q22	351	279	286	279
Total cold run time: 44239 ms
Total hot run time: 39766 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4375	4373	4384	4373
q2	267	167	163	163
q3	3522	3530	3524	3524
q4	2379	2369	2358	2358
q5	5720	5723	5729	5723
q6	239	120	125	120
q7	2419	1872	1858	1858
q8	3507	3534	3507	3507
q9	9025	8984	9001	8984
q10	3889	3997	4009	3997
q11	504	389	373	373
q12	768	606	605	605
q13	4264	3583	3573	3573
q14	284	252	260	252
q15	573	518	508	508
q16	500	453	472	453
q17	1883	1846	1845	1845
q18	8656	8316	8338	8316
q19	1727	1791	1740	1740
q20	2253	1943	1932	1932
q21	6465	6196	6130	6130
q22	508	414	430	414
Total cold run time: 63727 ms
Total hot run time: 60748 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.78 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17222701301 Bytes

@jacktengg jacktengg force-pushed the fix1216 branch 2 times, most recently from e69ab64 to bdc1ccb Compare December 16, 2023 15:14
@jacktengg
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit bdc1ccbb4c1836abde1d32469a45c0a90c1ca1ea, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4718	4488	4457	4457
q2	364	151	160	151
q3	1459	1254	1210	1210
q4	1115	911	903	903
q5	3179	3178	3183	3178
q6	246	128	124	124
q7	1000	482	486	482
q8	2180	2182	2205	2182
q9	6717	6695	6650	6650
q10	3198	3283	3250	3250
q11	328	205	205	205
q12	347	212	206	206
q13	4565	3788	3838	3788
q14	242	211	208	208
q15	569	521	523	521
q16	440	391	384	384
q17	1013	613	594	594
q18	7264	6864	7024	6864
q19	1554	1446	1387	1387
q20	557	313	291	291
q21	3043	2619	2768	2619
q22	348	281	294	281
Total cold run time: 44446 ms
Total hot run time: 39935 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4424	4380	4397	4380
q2	268	163	171	163
q3	3539	3524	3529	3524
q4	2382	2369	2358	2358
q5	5734	5714	5720	5714
q6	241	123	124	123
q7	2384	1826	1874	1826
q8	3521	3513	3523	3513
q9	8985	9005	8973	8973
q10	3923	4014	3999	3999
q11	492	369	375	369
q12	776	584	596	584
q13	4290	3543	3573	3543
q14	285	256	259	256
q15	583	512	530	512
q16	495	456	460	456
q17	1880	1846	1838	1838
q18	8687	8312	8287	8287
q19	1745	1722	1759	1722
q20	2276	1949	1931	1931
q21	6523	6181	6169	6169
q22	518	436	432	432
Total cold run time: 63951 ms
Total hot run time: 60672 ms

@jacktengg
Copy link
Contributor Author

run clickbench

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit bdc1ccbb4c1836abde1d32469a45c0a90c1ca1ea, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4767	4472	4468	4468
q2	359	155	158	155
q3	1463	1224	1237	1224
q4	1112	890	873	873
q5	3162	3176	3129	3129
q6	248	128	129	128
q7	980	485	497	485
q8	2210	2218	2210	2210
q9	6677	6646	6744	6646
q10	3209	3255	3277	3255
q11	330	203	203	203
q12	349	222	208	208
q13	4586	3823	3822	3822
q14	244	209	209	209
q15	569	525	522	522
q16	437	393	384	384
q17	1023	629	495	495
q18	7127	6893	6855	6855
q19	1541	1393	1381	1381
q20	549	312	310	310
q21	3116	2665	2671	2665
q22	352	289	290	289
Total cold run time: 44410 ms
Total hot run time: 39916 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4385	4404	4391	4391
q2	268	165	172	165
q3	3547	3515	3518	3515
q4	2379	2367	2363	2363
q5	5762	5740	5849	5740
q6	244	123	123	123
q7	2395	1857	1898	1857
q8	3525	3530	3519	3519
q9	9031	9018	9012	9012
q10	3912	3995	3999	3995
q11	522	382	373	373
q12	770	598	576	576
q13	4328	3539	3570	3539
q14	292	251	255	251
q15	575	517	523	517
q16	498	456	496	456
q17	1866	1847	1893	1847
q18	8636	8380	8366	8366
q19	1752	1865	1759	1759
q20	2273	1938	1930	1930
q21	6552	6192	6165	6165
q22	511	431	423	423
Total cold run time: 64023 ms
Total hot run time: 60882 ms

@jacktengg
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit cb64f948d7139be886b8c8b1595eb95e582d0ecc, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4685	4436	4454	4436
q2	362	152	162	152
q3	1458	1278	1234	1234
q4	1120	923	899	899
q5	3153	3153	3151	3151
q6	244	127	125	125
q7	983	482	473	473
q8	2179	2194	2201	2194
q9	6696	6680	6680	6680
q10	3196	3298	3253	3253
q11	321	201	197	197
q12	347	205	212	205
q13	4535	3784	3790	3784
q14	240	208	214	208
q15	572	527	513	513
q16	434	388	381	381
q17	1030	597	506	506
q18	7179	7038	6873	6873
q19	1544	1430	1465	1430
q20	515	319	315	315
q21	3086	2667	2651	2651
q22	352	284	291	284
Total cold run time: 44231 ms
Total hot run time: 39944 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4407	4382	4397	4382
q2	269	162	172	162
q3	3524	3512	3517	3512
q4	2380	2358	2349	2349
q5	5713	5705	5726	5705
q6	239	118	119	118
q7	2389	1891	1880	1880
q8	3502	3508	3502	3502
q9	8979	8891	8988	8891
q10	3899	3979	3971	3971
q11	509	373	381	373
q12	763	605	593	593
q13	4278	3534	3565	3534
q14	276	265	267	265
q15	560	523	520	520
q16	498	450	442	442
q17	1884	1848	1806	1806
q18	8669	8279	8253	8253
q19	1721	1703	1756	1703
q20	2246	1954	1926	1926
q21	6499	6148	6186	6148
q22	515	438	405	405
Total cold run time: 63719 ms
Total hot run time: 60440 ms

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@jacktengg jacktengg force-pushed the fix1216 branch 2 times, most recently from 42951b8 to aaf914c Compare December 17, 2023 15:09
@jacktengg
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit aaf914ca6ae0a5c9f1aa552b87589c586cba5ca9, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4688	4474	4445	4445
q2	359	164	160	160
q3	1450	1244	1230	1230
q4	1118	894	897	894
q5	3167	3140	3145	3140
q6	244	127	128	127
q7	999	488	495	488
q8	2213	2221	2196	2196
q9	6695	6695	6664	6664
q10	3235	3256	3275	3256
q11	333	201	202	201
q12	350	213	206	206
q13	4578	3823	3820	3820
q14	243	211	215	211
q15	579	525	521	521
q16	445	385	392	385
q17	1011	619	506	506
q18	7308	7026	6965	6965
q19	1537	1426	1374	1374
q20	553	307	294	294
q21	3072	2641	2656	2641
q22	350	274	283	274
Total cold run time: 44527 ms
Total hot run time: 39998 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4411	4389	4406	4389
q2	266	163	174	163
q3	3535	3518	3526	3518
q4	2378	2369	2366	2366
q5	5719	5736	5834	5736
q6	245	122	126	122
q7	2384	1921	1938	1921
q8	3533	3526	3528	3526
q9	9177	9069	9026	9026
q10	3938	4175	4021	4021
q11	501	388	391	388
q12	765	629	655	629
q13	4494	3561	3737	3561
q14	299	264	291	264
q15	593	521	530	521
q16	532	480	461	461
q17	1877	1891	1880	1880
q18	8723	8560	8444	8444
q19	1726	1761	1783	1761
q20	2356	1969	1942	1942
q21	6720	6175	6325	6175
q22	532	428	422	422
Total cold run time: 64704 ms
Total hot run time: 61236 ms

@jacktengg
Copy link
Contributor Author

run clickbench

yiguolei
yiguolei approved these changes Dec 18, 2023
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei
Copy link
Contributor

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 18, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit a07e3fd4e2d2fb3c7a3f5a2e7880041583477b15, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4787	4489	4482	4482
q2	369	145	159	145
q3	1469	1246	1240	1240
q4	1109	878	872	872
q5	3158	3156	3159	3156
q6	245	127	128	127
q7	1012	486	489	486
q8	2205	2208	2193	2193
q9	6733	6688	6670	6670
q10	3221	3275	3263	3263
q11	317	206	212	206
q12	351	214	214	214
q13	4550	3806	3785	3785
q14	240	210	210	210
q15	567	516	531	516
q16	439	385	396	385
q17	1016	655	553	553
q18	7137	6930	6964	6930
q19	1535	1471	1386	1386
q20	547	325	307	307
q21	3097	2601	2727	2601
q22	351	275	281	275
Total cold run time: 44455 ms
Total hot run time: 40002 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4412	4406	4406	4406
q2	272	162	172	162
q3	3545	3538	3518	3518
q4	2398	2374	2376	2374
q5	5731	5733	5734	5733
q6	242	123	121	121
q7	2381	1855	1906	1855
q8	3541	3542	3538	3538
q9	9060	8966	9026	8966
q10	3945	4018	4008	4008
q11	503	387	388	387
q12	773	611	608	608
q13	4297	3552	3563	3552
q14	289	247	256	247
q15	580	512	531	512
q16	502	450	453	450
q17	1890	1862	1854	1854
q18	8713	8302	8256	8256
q19	1741	1770	1775	1770
q20	2250	1938	1937	1937
q21	6550	6206	6184	6184
q22	495	424	426	424
Total cold run time: 64110 ms
Total hot run time: 60862 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.74 seconds
stream load tsv: 573 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17222680355 Bytes

@yiguolei yiguolei merged commit 9ebacb1 into apache:master Dec 18, 2023
25 of 26 checks passed
morningman pushed a commit that referenced this pull request Dec 19, 2023
@jacktengg @morningman
[fix](expr) fix performance problem caused by too many virtual functi…
2121c97 
…on call (#28508)
jacktengg added a commit to jacktengg/incubator-doris that referenced this pull request Dec 20, 2023
@jacktengg
[fix](expr) fix performance problem caused by too many virtual functi…
75231a9 
…on call (apache#28508)
@jacktengg jacktengg mentioned this pull request Dec 20, 2023
Merged
jacktengg added a commit to jacktengg/incubator-doris that referenced this pull request Dec 20, 2023
@jacktengg
[fix](expr) fix performance problem caused by too many virtual functi…
1052ad0 
…on call (apache#28508)
yiguolei pushed a commit that referenced this pull request Dec 20, 2023
@jacktengg
[fix](expr) fix performance problem caused by too many virtual functi…
8f175f9 
…on call (#28508) (#28689)

* [fix](expr) fix performance problem caused by too many virtual function call (#28508)

* fix gcc compile error
hello-stephen pushed a commit to hello-stephen/doris that referenced this pull request Dec 28, 2023
@jacktengg
[fix](expr) fix performance problem caused by too many virtual functi…
9d573a9 
…on call (apache#28508)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@jacktengg @HappenLee
[fix](expr) fix performance problem caused by too many virtual functi…
dc3fda4 
…on call (apache#28508)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiguolei yiguolei yiguolei approved these changes

@airborne12 airborne12 airborne12 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jacktengg @doris-robot @yiguolei @airborne12