Skip to content

[Function](opt) do base64 opt remove the unless mem alloc #32019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
HappenLee merged 1 commit into from
Mar 11, 2024
Merged

[Function](opt) do base64 opt remove the unless mem alloc #32019

HappenLee merged 1 commit into from
Mar 11, 2024

Conversation

@HappenLee
Copy link
Contributor

@HappenLee HappenLee commented Mar 8, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Mar 8, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@HappenLee
Copy link
Contributor Author

HappenLee commented Mar 8, 2024

run buildall

1 similar comment
@HappenLee
Copy link
Contributor Author

HappenLee commented Mar 9, 2024

run buildall

@HappenLee HappenLee changed the title [Function](lib) use libbase64 to do base64 [Function](opt) do base64 opt remove the unless mem alloc Mar 10, 2024
@HappenLee
Copy link
Contributor Author

HappenLee commented Mar 10, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 10, 2024

TeamCity be ut coverage result:
Function Coverage: 35.35% (8553/24195)
Line Coverage: 27.13% (69490/256127)
Region Coverage: 26.35% (36052/136831)
Branch Coverage: 23.20% (18428/79438)
Coverage Report: http://coverage.selectdb-in.cc/coverage/66d444e9ec35e664ac6e34997d2a39452cd519f0_66d444e9ec35e664ac6e34997d2a39452cd519f0/report/index.html

@doris-robot
Copy link

doris-robot commented Mar 10, 2024

TPC-H: Total hot run time: 38176 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66d444e9ec35e664ac6e34997d2a39452cd519f0, data reload: false

------ Round 1 ----------------------------------
q1	18533	4151	4139	4139
q2	2653	178	142	142
q3	11162	1007	958	958
q4	5291	967	964	964
q5	7732	2942	3015	2942
q6	179	130	126	126
q7	1311	821	804	804
q8	9527	2051	2070	2051
q9	7265	6504	6489	6489
q10	8263	2557	2543	2543
q11	441	216	218	216
q12	805	324	311	311
q13	17949	2999	2913	2913
q14	279	255	246	246
q15	484	443	439	439
q16	478	389	388	388
q17	952	870	862	862
q18	6636	5962	5879	5879
q19	1589	1525	1527	1525
q20	542	293	291	291
q21	7492	3723	3658	3658
q22	798	318	290	290
Total cold run time: 110361 ms
Total hot run time: 38176 ms

----- Round 2, with runtime_filter_mode=off -----
q1	3999	4000	4050	4000
q2	319	232	223	223
q3	2950	2921	2935	2921
q4	1853	1810	1793	1793
q5	5248	5249	5232	5232
q6	210	122	118	118
q7	2241	1817	1848	1817
q8	3224	3279	3290	3279
q9	8558	8548	8536	8536
q10	6101	3747	3693	3693
q11	524	434	445	434
q12	694	552	532	532
q13	14468	2790	2784	2784
q14	270	248	257	248
q15	466	441	431	431
q16	437	389	399	389
q17	1682	1672	1665	1665
q18	7777	7507	7237	7237
q19	1620	1614	1598	1598
q20	1970	1738	1731	1731
q21	4887	4777	4741	4741
q22	544	468	487	468
Total cold run time: 70042 ms
Total hot run time: 53870 ms

@doris-robot
Copy link

doris-robot commented Mar 10, 2024

TPC-DS: Total hot run time: 186146 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 66d444e9ec35e664ac6e34997d2a39452cd519f0, data reload: false

query1	924	360	334	334
query2	7353	2041	2028	2028
query3	6713	212	211	211
query4	25886	20832	21157	20832
query5	4296	539	444	444
query6	282	186	187	186
query7	4636	327	309	309
query8	242	181	171	171
query9	8337	2254	2239	2239
query10	449	237	252	237
query11	14918	14352	14334	14334
query12	135	93	86	86
query13	1642	413	435	413
query14	10440	10121	9626	9626
query15	239	198	188	188
query16	7811	285	283	283
query17	1285	589	582	582
query18	1975	316	289	289
query19	205	167	163	163
query20	92	87	90	87
query21	204	127	127	127
query22	4698	4410	4414	4410
query23	31619	31088	30920	30920
query24	12221	3084	3089	3084
query25	704	390	384	384
query26	1926	174	171	171
query27	2959	370	356	356
query28	6667	1859	1831	1831
query29	1178	635	611	611
query30	316	151	154	151
query31	934	743	747	743
query32	102	62	59	59
query33	757	255	270	255
query34	968	474	480	474
query35	869	652	626	626
query36	928	815	809	809
query37	282	76	78	76
query38	3258	3117	3118	3117
query39	1411	1366	1372	1366
query40	306	120	115	115
query41	63	58	58	58
query42	111	100	108	100
query43	442	405	414	405
query44	1086	721	715	715
query45	272	254	260	254
query46	1075	831	786	786
query47	1619	1533	1572	1533
query48	418	351	353	351
query49	1251	380	327	327
query50	795	379	386	379
query51	6656	6465	6538	6465
query52	105	94	94	94
query53	350	287	289	287
query54	350	252	251	251
query55	87	84	86	84
query56	244	230	228	228
query57	1086	984	995	984
query58	253	220	247	220
query59	2496	2463	2347	2347
query60	264	255	262	255
query61	136	115	117	115
query62	653	434	411	411
query63	314	282	288	282
query64	6301	3310	3036	3036
query65	3028	3038	3001	3001
query66	1394	335	326	326
query67	15030	14333	14363	14333
query68	12957	561	565	561
query69	724	382	403	382
query70	1458	1097	1072	1072
query71	627	272	271	271
query72	9943	2611	2491	2491
query73	3303	345	345	345
query74	7216	6811	6787	6787
query75	12515	8077	8062	8062
query76	8050	1100	1169	1100
query77	902	250	273	250
query78	10195	9725	9543	9543
query79	11141	517	506	506
query80	924	384	379	379
query81	492	204	212	204
query82	344	189	196	189
query83	261	143	142	142
query84	284	74	76	74
query85	1192	344	326	326
query86	372	304	301	301
query87	3396	3164	3166	3164
query88	3126	2355	2346	2346
query89	535	371	363	363
query90	2335	173	177	173
query91	162	127	153	127
query92	64	47	48	47
query93	4802	542	528	528
query94	1593	187	196	187
query95	442	349	347	347
query96	586	265	263	263
query97	4456	4278	4286	4278
query98	226	232	236	232
query99	1105	775	752	752
Total cold run time: 321774 ms
Total hot run time: 186146 ms

@doris-robot
Copy link

doris-robot commented Mar 10, 2024

ClickBench: Total hot run time: 30.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 66d444e9ec35e664ac6e34997d2a39452cd519f0, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.68	0.10	0.10
query5	0.48	0.49	0.52
query6	1.27	0.67	0.66
query7	0.02	0.02	0.02
query8	0.04	0.02	0.02
query9	0.54	0.53	0.53
query10	0.56	0.56	0.58
query11	0.13	0.09	0.10
query12	0.12	0.10	0.10
query13	0.59	0.57	0.57
query14	0.75	0.74	0.74
query15	0.83	0.79	0.80
query16	0.37	0.38	0.36
query17	0.99	0.97	0.93
query18	0.25	0.25	0.25
query19	1.82	1.70	1.71
query20	0.01	0.01	0.01
query21	15.41	0.63	0.64
query22	3.41	3.68	1.92
query23	17.26	1.02	0.99
query24	2.33	0.38	0.38
query25	0.24	0.05	0.05
query26	0.20	0.14	0.13
query27	0.03	0.03	0.03
query28	12.18	0.85	0.86
query29	12.56	3.34	3.43
query30	0.66	0.57	0.57
query31	2.78	0.34	0.34
query32	3.39	0.43	0.43
query33	2.89	2.89	2.90
query34	15.50	4.34	4.32
query35	4.31	4.34	4.32
query36	1.09	1.01	1.02
query37	0.07	0.05	0.05
query38	0.05	0.03	0.03
query39	0.02	0.02	0.02
query40	0.19	0.13	0.14
query41	0.08	0.02	0.02
query42	0.02	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.48 s
Total hot run time: 30.52 s

@doris-robot
Copy link

doris-robot commented Mar 10, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 66d444e9ec35e664ac6e34997d2a39452cd519f0 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       17.3 seconds inserted 10000000 Rows, about 578K ops/s

@github-actions
Copy link
Contributor

github-actions bot commented Mar 11, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 11, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Mar 11, 2024

PR approved by anyone and no changes requested.

@HappenLee HappenLee merged commit 3381eb8 into apache:master Mar 11, 2024
yiguolei pushed a commit that referenced this pull request Mar 14, 2024
@HappenLee @Doris-Extras
[Opt](exec) remove the unless mem alloc in base64 (#32019)
f2404ff
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Gabriel39 Gabriel39 Gabriel39 approved these changes

@zhangstar333 zhangstar333 zhangstar333 approved these changes

+1 more reviewer

@Mryange Mryange Mryange approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@HappenLee @doris-robot @Gabriel39 @Mryange @zhangstar333 @yiguolei