Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](memory) Modify memory gc conf and add crash_in_alloc_large_memory_bytes #39611

Merged
merged 1 commit into from
Aug 21, 2024
Merged

[opt](memory) Modify memory gc conf and add crash_in_alloc_large_memory_bytes #39611

merged 1 commit into from
Aug 21, 2024

Conversation

xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Aug 20, 2024 •
edited
Loading

  1. faster frequency and small batch memory GC, which helps query stability.
  2. when alloc memory larger than crash_in_alloc_large_memory_bytes will crash, default -1 means disabled. if you need a core dump to analyze large memory allocation, modify this parameter to crash when large memory allocation occur will help

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 20, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 38186 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4d2ed3bca94e7ddeedeb80f175c1a2d1a073620f, data reload: false

------ Round 1 ----------------------------------
q1	18485	4736	4372	4372
q2	2072	205	215	205
q3	11322	983	1094	983
q4	10512	793	781	781
q5	7797	2887	2814	2814
q6	270	161	159	159
q7	1035	680	664	664
q8	9606	2154	2091	2091
q9	8828	6572	6577	6572
q10	7074	2231	2287	2231
q11	491	279	267	267
q12	438	259	260	259
q13	18875	3020	3010	3010
q14	300	263	257	257
q15	562	533	541	533
q16	533	436	406	406
q17	1012	718	699	699
q18	7459	6857	6685	6685
q19	6161	1082	967	967
q20	684	356	345	345
q21	3790	2883	2858	2858
q22	1164	1039	1028	1028
Total cold run time: 118470 ms
Total hot run time: 38186 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4558	4295	4319	4295
q2	390	315	309	309
q3	2880	2633	2738	2633
q4	1996	1670	1693	1670
q5	5636	5756	5618	5618
q6	232	153	147	147
q7	2141	1749	1772	1749
q8	3342	3500	3477	3477
q9	8769	8731	8756	8731
q10	3657	3349	3233	3233
q11	621	513	533	513
q12	824	655	656	655
q13	15830	3124	3208	3124
q14	327	290	284	284
q15	559	517	510	510
q16	487	445	432	432
q17	1860	1549	1551	1549
q18	8356	7953	7747	7747
q19	5878	1519	1647	1519
q20	2232	1888	1892	1888
q21	13815	5373	5345	5345
q22	1185	1098	1097	1097
Total cold run time: 85575 ms
Total hot run time: 56525 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196974 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4d2ed3bca94e7ddeedeb80f175c1a2d1a073620f, data reload: false

query1	1296	897	877	877
query2	6615	1988	1948	1948
query3	10625	3886	3726	3726
query4	56336	28649	23228	23228
query5	5570	648	653	648
query6	523	194	193	193
query7	6306	331	327	327
query8	477	397	430	397
query9	8442	2572	2509	2509
query10	524	342	336	336
query11	17829	14964	15369	14964
query12	202	140	146	140
query13	1654	465	479	465
query14	11428	7347	7592	7347
query15	235	198	191	191
query16	7191	520	553	520
query17	1186	617	630	617
query18	1869	354	343	343
query19	229	176	171	171
query20	151	142	137	137
query21	248	143	146	143
query22	4656	4634	4614	4614
query23	34158	33689	33743	33689
query24	5698	3020	3051	3020
query25	552	435	431	431
query26	716	190	180	180
query27	1923	313	309	309
query28	4090	2199	2180	2180
query29	699	448	448	448
query30	235	194	190	190
query31	1031	828	806	806
query32	94	78	79	78
query33	576	347	348	347
query34	905	495	505	495
query35	905	736	780	736
query36	1077	958	926	926
query37	162	103	105	103
query38	3885	3908	3849	3849
query39	1534	1480	1487	1480
query40	237	182	150	150
query41	139	137	136	136
query42	139	120	115	115
query43	541	501	489	489
query44	1106	780	779	779
query45	233	197	192	192
query46	1135	829	781	781
query47	1924	1826	1835	1826
query48	402	334	328	328
query49	905	602	587	587
query50	866	460	468	460
query51	7265	7141	7144	7141
query52	126	108	114	108
query53	299	228	230	228
query54	598	508	496	496
query55	94	90	88	88
query56	334	314	318	314
query57	1198	1093	1136	1093
query58	293	377	308	308
query59	2981	2793	2868	2793
query60	341	328	326	326
query61	155	147	145	145
query62	782	676	692	676
query63	254	229	227	227
query64	2716	1851	1853	1851
query65	3258	3176	3190	3176
query66	1012	665	674	665
query67	15340	14941	14890	14890
query68	6798	593	595	593
query69	731	422	361	361
query70	1243	1185	1184	1184
query71	542	320	313	313
query72	6774	2304	2046	2046
query73	823	353	351	351
query74	9304	8840	8953	8840
query75	3760	2739	2771	2739
query76	3817	993	980	980
query77	832	440	441	440
query78	10069	10096	9611	9611
query79	7393	557	560	557
query80	1386	616	625	616
query81	610	269	264	264
query82	518	167	167	167
query83	365	214	216	214
query84	295	102	94	94
query85	810	360	351	351
query86	473	319	335	319
query87	4479	4221	4199	4199
query88	4553	2457	2487	2457
query89	492	330	319	319
query90	1909	231	230	230
query91	153	130	126	126
query92	86	74	74	74
query93	3145	550	548	548
query94	785	335	362	335
query95	383	291	290	290
query96	611	289	285	285
query97	3287	3083	3056	3056
query98	247	245	226	226
query99	1756	1313	1323	1313
Total cold run time: 324934 ms
Total hot run time: 196974 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.02 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4d2ed3bca94e7ddeedeb80f175c1a2d1a073620f, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.70	0.08	0.08
query5	0.50	0.48	0.49
query6	1.15	0.75	0.73
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.56	0.49	0.49
query10	0.55	0.53	0.54
query11	0.16	0.12	0.11
query12	0.15	0.13	0.13
query13	0.61	0.59	0.60
query14	0.77	0.79	0.78
query15	0.90	0.82	0.81
query16	0.38	0.39	0.37
query17	0.98	0.97	1.06
query18	0.22	0.20	0.20
query19	1.82	1.84	1.70
query20	0.01	0.02	0.01
query21	15.40	0.66	0.65
query22	4.01	7.60	1.97
query23	18.32	1.44	1.31
query24	2.14	0.22	0.23
query25	0.15	0.09	0.08
query26	0.27	0.18	0.18
query27	0.09	0.08	0.08
query28	13.22	1.04	1.00
query29	12.63	3.31	3.35
query30	0.44	0.25	0.24
query31	2.82	0.41	0.40
query32	3.24	0.49	0.49
query33	2.95	2.98	3.00
query34	17.07	4.32	4.38
query35	4.43	4.41	4.47
query36	0.68	0.48	0.48
query37	0.21	0.17	0.17
query38	0.17	0.16	0.17
query39	0.07	0.05	0.06
query40	0.19	0.15	0.16
query41	0.11	0.06	0.08
query42	0.09	0.08	0.08
query43	0.08	0.07	0.07
Total cold run time: 109.68 s
Total hot run time: 31.02 s

dataroaring
dataroaring approved these changes Aug 21, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 6ec299e into apache:master Aug 21, 2024
28 of 32 checks passed
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Aug 23, 2024
@xinyiZzz
[opt](memory) Modify memory gc conf and add `crash_in_alloc_large_mem…
4cb9811 
…ory_bytes` (apache#39611)

1. faster frequency and small batch memory GC, which helps query
stability.
2. when alloc memory larger than crash_in_alloc_large_memory_bytes will
crash, default -1 means disabled. if you need a core dump to analyze
large memory allocation, modify this parameter to crash when large
memory allocation occur will help
@xinyiZzz xinyiZzz mentioned this pull request Aug 23, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 24, 2024
@xinyiZzz
[branch-2.1](memory) Modify memory gc conf and add `crash_in_alloc_la…
ae4d747 
…rge_memory_bytes` (#39834)

pick #39611
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@xinyiZzz @dataroaring
[opt](memory) Modify memory gc conf and add `crash_in_alloc_large_mem…
dd8153a 
…ory_bytes` (#39611)

1. faster frequency and small batch memory GC, which helps query
stability.
2. when alloc memory larger than crash_in_alloc_large_memory_bytes will
crash, default -1 means disabled. if you need a core dump to analyze
large memory allocation, modify this parameter to crash when large
memory allocation occur will help
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@xinyiZzz @doris-robot @yiguolei @dataroaring