Skip to content

[Improvement](memory) clear arena when finalize one row#30788

Merged
yiguolei merged 1 commit into
apache:masterfrom
BiteTheDDDDt:dev_0204
Feb 7, 2024
Merged

[Improvement](memory) clear arena when finalize one row#30788
yiguolei merged 1 commit into
apache:masterfrom
BiteTheDDDDt:dev_0204

Conversation

@BiteTheDDDDt
Copy link
Copy Markdown
Contributor

@BiteTheDDDDt BiteTheDDDDt commented Feb 4, 2024

Proposed changes

clear arena when finalize one row

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 4, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@BiteTheDDDDt
Copy link
Copy Markdown
Contributor Author

BiteTheDDDDt commented Feb 4, 2024

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 4, 2024

TPC-H: Total hot run time: 36780 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b68ea417fae677f99f929c2971152bdb6a1ac704, data reload: false

------ Round 1 ----------------------------------
q1	17721	4589	4434	4434
q2	2037	139	132	132
q3	10682	958	893	893
q4	4652	747	687	687
q5	7721	2758	2792	2758
q6	185	118	120	118
q7	1155	726	717	717
q8	9315	2005	2014	2005
q9	7545	6469	6453	6453
q10	8121	2432	2423	2423
q11	414	207	207	207
q12	741	267	280	267
q13	18278	3323	3375	3323
q14	292	255	262	255
q15	544	484	493	484
q16	483	413	420	413
q17	940	554	534	534
q18	6890	5936	5852	5852
q19	1549	1407	1227	1227
q20	627	325	344	325
q21	6631	3054	2977	2977
q22	817	300	296	296
Total cold run time: 107340 ms
Total hot run time: 36780 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4489	4360	4349	4349
q2	330	227	230	227
q3	2979	2884	2831	2831
q4	1788	1593	1590	1590
q5	5211	5258	5241	5241
q6	188	115	115	115
q7	2171	1759	1795	1759
q8	3128	3258	3288	3258
q9	8354	8300	8279	8279
q10	5824	3565	3509	3509
q11	545	468	451	451
q12	770	586	584	584
q13	12737	3077	3043	3043
q14	268	234	258	234
q15	532	483	494	483
q16	521	467	459	459
q17	1839	1677	1679	1677
q18	7941	7796	7631	7631
q19	7057	1522	1548	1522
q20	2118	1940	1915	1915
q21	4734	4555	4634	4555
q22	543	457	452	452
Total cold run time: 74067 ms
Total hot run time: 54164 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 4, 2024

TeamCity be ut coverage result:
Function Coverage: 36.16% (8626/23858)
Line Coverage: 28.22% (70544/250009)
Region Coverage: 27.22% (36394/133684)
Branch Coverage: 24.01% (18641/77624)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b68ea417fae677f99f929c2971152bdb6a1ac704_b68ea417fae677f99f929c2971152bdb6a1ac704/report/index.html

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 4, 2024

TPC-DS: Total hot run time: 175040 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b68ea417fae677f99f929c2971152bdb6a1ac704, data reload: false

query1	932	344	330	330
query2	6560	1993	1910	1910
query3	6731	209	204	204
query4	31775	22129	22196	22129
query5	4291	363	418	363
query6	257	166	171	166
query7	4601	287	276	276
query8	245	190	182	182
query9	8920	2361	2335	2335
query10	419	243	216	216
query11	16940	15573	15607	15573
query12	142	80	89	80
query13	1738	433	443	433
query14	10396	7274	7154	7154
query15	272	188	198	188
query16	8247	275	253	253
query17	2197	571	523	523
query18	2368	275	263	263
query19	354	144	142	142
query20	85	76	79	76
query21	194	139	118	118
query22	4915	4705	4536	4536
query23	30785	30067	30040	30040
query24	10493	2773	2752	2752
query25	571	344	341	341
query26	1160	139	140	139
query27	2958	295	310	295
query28	7598	1865	1873	1865
query29	869	618	598	598
query30	279	132	139	132
query31	917	712	725	712
query32	88	53	46	46
query33	729	223	221	221
query34	1098	455	464	455
query35	850	783	747	747
query36	1070	918	940	918
query37	117	57	60	57
query38	3189	3115	3117	3115
query39	1307	1239	1244	1239
query40	190	97	94	94
query41	40	35	35	35
query42	100	89	93	89
query43	537	481	504	481
query44	1063	696	705	696
query45	189	182	175	175
query46	1063	654	650	650
query47	1489	1462	1515	1462
query48	461	352	348	348
query49	1084	294	283	283
query50	762	381	385	381
query51	5191	5125	5152	5125
query52	100	82	92	82
query53	356	264	273	264
query54	272	218	221	218
query55	84	72	77	72
query56	219	217	196	196
query57	977	911	878	878
query58	209	176	184	176
query59	2324	2260	2266	2260
query60	252	211	221	211
query61	85	88	83	83
query62	652	373	363	363
query63	293	258	282	258
query64	5266	3707	3470	3470
query65	3234	3224	3234	3224
query66	815	319	319	319
query67	14622	14359	14163	14163
query68	5334	536	545	536
query69	462	321	338	321
query70	1270	1271	1205	1205
query71	335	256	252	252
query72	6365	2856	2703	2703
query73	697	318	317	317
query74	6611	6103	6256	6103
query75	3058	2336	2352	2336
query76	3270	857	954	857
query77	475	226	226	226
query78	9212	8784	8522	8522
query79	2398	505	507	505
query80	1741	359	354	354
query81	541	193	196	193
query82	653	80	80	80
query83	281	127	132	127
query84	250	81	80	80
query85	2164	346	325	325
query86	484	293	300	293
query87	3384	3222	3246	3222
query88	3757	2371	2350	2350
query89	436	374	388	374
query90	2094	166	165	165
query91	169	120	138	120
query92	60	44	42	42
query93	2261	497	471	471
query94	1449	174	172	172
query95	478	366	8006	366
query96	598	270	278	270
query97	4273	4133	4097	4097
query98	229	206	184	184
query99	1153	694	699	694
Total cold run time: 284528 ms
Total hot run time: 175040 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 4, 2024

ClickBench: Total hot run time: 30.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b68ea417fae677f99f929c2971152bdb6a1ac704, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.02	0.02
query3	0.22	0.06	0.06
query4	1.68	0.10	0.10
query5	0.53	0.53	0.51
query6	1.18	0.64	0.67
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.55	0.56	0.55
query11	0.12	0.09	0.09
query12	0.11	0.08	0.10
query13	0.60	0.62	0.61
query14	0.80	0.80	0.79
query15	0.80	0.78	0.78
query16	0.39	0.37	0.38
query17	1.03	0.99	1.00
query18	0.24	0.20	0.21
query19	1.83	1.73	1.79
query20	0.01	0.01	0.02
query21	15.40	0.60	0.58
query22	2.52	2.66	2.30
query23	17.38	0.69	0.80
query24	2.48	0.71	0.46
query25	0.37	0.17	0.07
query26	0.45	0.14	0.14
query27	0.05	0.04	0.05
query28	12.26	0.83	0.82
query29	12.52	3.11	3.19
query30	0.60	0.55	0.47
query31	2.79	0.34	0.34
query32	3.37	0.48	0.48
query33	3.21	3.21	3.23
query34	15.79	4.31	4.29
query35	4.31	4.30	4.34
query36	1.11	1.05	1.06
query37	0.06	0.05	0.05
query38	0.05	0.02	0.03
query39	0.02	0.02	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.01
query42	0.02	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 105.82 s
Total hot run time: 30.71 s

@doris-robot
Copy link
Copy Markdown

doris-robot commented Feb 4, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b68ea417fae677f99f929c2971152bdb6a1ac704 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.5 seconds inserted 10000000 Rows, about 689K ops/s

@zhannngchen zhannngchen requested a review from liaoxin01 February 7, 2024 03:33
yiguolei
yiguolei approved these changes Feb 7, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 7, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Feb 7, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 7, 2024

PR approved by anyone and no changes requested.

@yiguolei
Copy link
Copy Markdown
Contributor

yiguolei commented Feb 7, 2024

Very very important work, This PR will reduce many memory for agg table.

@yiguolei yiguolei merged commit 70e2a40 into apache:master Feb 7, 2024
yiguolei pushed a commit that referenced this pull request Feb 7, 2024
@BiteTheDDDDt @Doris-Extras
[Improvement](memory) clear arena when finalize one row #30788
38c72f0
zhannngchen pushed a commit to zhannngchen/incubator-doris that referenced this pull request Feb 7, 2024
@BiteTheDDDDt @zhannngchen
[Improvement](memory) clear arena when finalize one row apache#30788
c6441d9
yiguolei pushed a commit that referenced this pull request Feb 12, 2024
@BiteTheDDDDt @Doris-Extras
[Improvement](memory) clear arena when finalize one row #30788
5f5542c
yiguolei pushed a commit that referenced this pull request Feb 16, 2024
@BiteTheDDDDt @Doris-Extras
[Improvement](memory) clear arena when finalize one row #30788
e96b3db
mymeiyi pushed a commit to mymeiyi/doris that referenced this pull request Feb 19, 2024
@BiteTheDDDDt @mymeiyi
[Improvement](memory) clear arena when finalize one row apache#30788
0702cbc
@BiteTheDDDDt BiteTheDDDDt deleted the dev_0204 branch January 20, 2025 06:52
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@BiteTheDDDDt @HappenLee
[Improvement](memory) clear arena when finalize one row apache#30788
014d8b6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@w41ter w41ter w41ter approved these changes

@yiguolei yiguolei yiguolei approved these changes

@liaoxin01 liaoxin01 Awaiting requested review from liaoxin01

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@BiteTheDDDDt @doris-robot @yiguolei @w41ter