Skip to content

[test](memory) Test query memory leak#33175

Closed
xinyiZzz wants to merge 3 commits intoapache:masterfrom
xinyiZzz:20240402_fix_memtracker_test
Closed

[test](memory) Test query memory leak#33175
xinyiZzz wants to merge 3 commits intoapache:masterfrom
xinyiZzz:20240402_fix_memtracker_test

Conversation

@xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Apr 2, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Apr 2, 2024

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@doris-robot
Copy link

TPC-H: Total hot run time: 38768 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a730cbeb87b4f6ba9a8a4454c0221840941abc57, data reload: false

------ Round 1 ----------------------------------
q1	17608	4188	4118	4118
q2	2017	194	183	183
q3	10470	1248	1404	1248
q4	10207	897	985	897
q5	7507	3019	2950	2950
q6	218	135	133	133
q7	1100	632	611	611
q8	9397	2004	2039	2004
q9	6659	6284	6120	6120
q10	8470	3510	3520	3510
q11	422	252	236	236
q12	385	217	213	213
q13	17775	2866	2911	2866
q14	282	241	246	241
q15	535	488	491	488
q16	501	404	383	383
q17	956	932	932	932
q18	7296	6590	6379	6379
q19	1613	1538	1681	1538
q20	608	322	313	313
q21	3568	3213	3099	3099
q22	373	306	308	306
Total cold run time: 107967 ms
Total hot run time: 38768 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4093	4061	4091	4061
q2	332	224	223	223
q3	2985	2970	2984	2970
q4	1891	1896	1917	1896
q5	5272	5242	5246	5242
q6	210	126	126	126
q7	2253	1841	1810	1810
q8	3227	3296	3291	3291
q9	8460	8447	8443	8443
q10	3800	4016	4057	4016
q11	560	480	457	457
q12	787	610	608	608
q13	13436	3079	3056	3056
q14	334	288	324	288
q15	545	495	513	495
q16	508	442	445	442
q17	1764	1717	1778	1717
q18	8389	7876	7579	7579
q19	1726	1670	1696	1670
q20	2039	1870	1881	1870
q21	5165	5099	5108	5099
q22	513	433	439	433
Total cold run time: 68289 ms
Total hot run time: 55792 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.65% (8882/24914)
Line Coverage: 27.37% (72908/266333)
Region Coverage: 26.56% (37714/142000)
Branch Coverage: 23.34% (19217/82322)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a730cbeb87b4f6ba9a8a4454c0221840941abc57_a730cbeb87b4f6ba9a8a4454c0221840941abc57/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 182349 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a730cbeb87b4f6ba9a8a4454c0221840941abc57, data reload: false

query1	1217	1120	1117	1117
query2	6278	2046	1848	1848
query3	6652	219	209	209
query4	24224	21534	21454	21454
query5	4177	406	407	406
query6	275	208	192	192
query7	4607	313	298	298
query8	228	173	191	173
query9	8460	2320	2309	2309
query10	453	257	266	257
query11	14986	14443	14504	14443
query12	141	98	95	95
query13	1630	394	375	375
query14	8520	7003	6933	6933
query15	214	177	184	177
query16	6701	286	288	286
query17	965	603	579	579
query18	1822	288	282	282
query19	199	163	168	163
query20	98	98	96	96
query21	200	132	132	132
query22	4907	4780	4794	4780
query23	33446	32999	32820	32820
query24	12839	3268	3281	3268
query25	739	440	464	440
query26	1961	178	172	172
query27	3274	387	403	387
query28	6994	1974	1908	1908
query29	1311	651	616	616
query30	317	179	176	176
query31	1031	783	771	771
query32	98	69	62	62
query33	713	265	273	265
query34	1198	521	551	521
query35	861	767	759	759
query36	1025	867	869	867
query37	183	96	97	96
query38	3722	3708	3619	3619
query39	1074	1078	1040	1040
query40	241	143	150	143
query41	49	50	45	45
query42	123	112	107	107
query43	450	410	404	404
query44	1199	730	755	730
query45	292	262	292	262
query46	1111	850	811	811
query47	1966	1845	1865	1845
query48	402	332	326	326
query49	916	376	387	376
query50	851	426	427	426
query51	6868	6815	6754	6754
query52	122	104	108	104
query53	366	307	309	307
query54	319	280	260	260
query55	94	92	82	82
query56	258	239	260	239
query57	1294	1186	1194	1186
query58	247	229	240	229
query59	2691	2440	2348	2348
query60	267	239	233	233
query61	89	91	89	89
query62	658	445	447	445
query63	313	289	289	289
query64	5727	3358	3165	3165
query65	3058	3005	3024	3005
query66	1321	312	321	312
query67	15463	14737	14996	14737
query68	9274	567	589	567
query69	596	329	333	329
query70	1411	1105	1124	1105
query71	487	273	274	273
query72	6399	2581	2393	2393
query73	1580	327	332	327
query74	6916	6341	6314	6314
query75	3488	2345	2291	2291
query76	5203	1126	1215	1126
query77	555	250	249	249
query78	11122	10141	10321	10141
query79	9760	551	549	549
query80	1462	456	451	451
query81	512	238	239	238
query82	427	110	109	109
query83	220	164	166	164
query84	266	92	91	91
query85	928	308	286	286
query86	363	274	289	274
query87	3710	3502	3489	3489
query88	3909	2372	2361	2361
query89	550	376	379	376
query90	1985	178	185	178
query91	134	119	107	107
query92	67	52	55	52
query93	6593	533	527	527
query94	1239	198	196	196
query95	439	362	333	333
query96	610	282	269	269
query97	2629	2492	2506	2492
query98	232	222	213	213
query99	1242	833	829	829
Total cold run time: 298803 ms
Total hot run time: 182349 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.9 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a730cbeb87b4f6ba9a8a4454c0221840941abc57, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.08
query5	0.49	0.47	0.50
query6	1.12	0.65	0.66
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.56	0.51	0.51
query10	0.56	0.57	0.56
query11	0.16	0.12	0.12
query12	0.14	0.11	0.12
query13	0.60	0.59	0.59
query14	0.77	0.78	0.80
query15	0.86	0.85	0.85
query16	0.36	0.36	0.35
query17	1.00	1.01	0.98
query18	0.26	0.25	0.25
query19	1.81	1.72	1.69
query20	0.02	0.01	0.01
query21	15.45	0.77	0.71
query22	2.85	5.33	2.48
query23	17.62	1.33	1.13
query24	1.35	0.24	0.41
query25	0.14	0.09	0.08
query26	0.29	0.18	0.19
query27	0.08	0.08	0.08
query28	13.42	0.97	0.95
query29	12.77	3.28	3.36
query30	0.27	0.07	0.06
query31	2.86	0.39	0.38
query32	3.27	0.47	0.47
query33	2.87	2.90	2.87
query34	15.49	4.32	4.34
query35	4.37	4.36	4.37
query36	0.67	0.47	0.48
query37	0.22	0.17	0.18
query38	0.18	0.16	0.16
query39	0.05	0.04	0.04
query40	0.18	0.15	0.15
query41	0.10	0.06	0.05
query42	0.06	0.05	0.05
query43	0.06	0.05	0.05
Total cold run time: 105.44 s
Total hot run time: 30.9 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a730cbeb87b4f6ba9a8a4454c0221840941abc57 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       15.6 seconds inserted 10000000 Rows, about 641K ops/s

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Apr 3, 2024

run buildall

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Apr 3, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Apr 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39249 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0c3b8d301420e66b02ce00a486a5ef5814c75ce5, data reload: false

------ Round 1 ----------------------------------
q1	17636	4197	4083	4083
q2	2018	199	188	188
q3	10469	1216	1428	1216
q4	10467	925	1056	925
q5	7735	3139	3026	3026
q6	226	136	135	135
q7	1160	673	650	650
q8	10506	2117	2109	2109
q9	6803	6341	6265	6265
q10	8496	3562	3551	3551
q11	420	247	240	240
q12	385	219	215	215
q13	17791	2921	2915	2915
q14	279	244	249	244
q15	536	499	482	482
q16	513	392	382	382
q17	991	931	928	928
q18	7354	6571	6410	6410
q19	1610	1533	1553	1533
q20	609	315	303	303
q21	3576	3202	3141	3141
q22	361	308	315	308
Total cold run time: 109941 ms
Total hot run time: 39249 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4070	4077	4060	4060
q2	327	212	221	212
q3	2986	2970	2962	2962
q4	1890	1861	1863	1861
q5	5267	5232	5284	5232
q6	207	124	123	123
q7	2271	1830	1819	1819
q8	3230	3340	3305	3305
q9	8545	8504	8485	8485
q10	3807	3833	3847	3833
q11	546	479	454	454
q12	731	540	583	540
q13	14520	2922	2903	2903
q14	297	253	274	253
q15	514	486	470	470
q16	470	416	400	400
q17	1741	1694	1705	1694
q18	7747	7351	7398	7351
q19	1664	1652	1657	1652
q20	1963	1731	1718	1718
q21	5023	4798	4821	4798
q22	505	415	439	415
Total cold run time: 68321 ms
Total hot run time: 54540 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 180836 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0c3b8d301420e66b02ce00a486a5ef5814c75ce5, data reload: false

query1	896	1109	1126	1109
query2	6471	1951	1897	1897
query3	6659	214	217	214
query4	24298	21554	21507	21507
query5	4204	416	404	404
query6	276	189	188	188
query7	4608	301	303	301
query8	231	167	177	167
query9	8462	2283	2272	2272
query10	598	269	254	254
query11	14968	14283	14833	14283
query12	149	100	98	98
query13	1653	385	386	385
query14	8614	6923	6723	6723
query15	226	179	180	179
query16	7174	284	277	277
query17	989	607	592	592
query18	1912	288	282	282
query19	202	155	159	155
query20	97	97	91	91
query21	200	124	126	124
query22	4944	4789	4769	4769
query23	33575	32940	32543	32543
query24	12270	3108	3152	3108
query25	691	383	390	383
query26	1889	161	174	161
query27	2992	331	323	323
query28	6782	1865	1833	1833
query29	1384	577	590	577
query30	300	170	165	165
query31	977	731	713	713
query32	98	60	57	57
query33	717	249	248	248
query34	1043	496	497	496
query35	838	726	715	715
query36	989	884	908	884
query37	290	81	80	80
query38	3553	3468	3376	3376
query39	1583	1537	1546	1537
query40	297	134	137	134
query41	49	48	47	47
query42	111	108	101	101
query43	451	406	409	406
query44	1085	723	715	715
query45	279	264	252	252
query46	1083	784	783	783
query47	1888	1791	1812	1791
query48	384	314	302	302
query49	1157	369	375	369
query50	788	387	393	387
query51	6729	6629	6539	6539
query52	113	97	100	97
query53	371	290	287	287
query54	322	240	243	240
query55	92	86	80	80
query56	257	231	226	226
query57	1215	1122	1130	1122
query58	254	228	226	226
query59	2499	2300	2591	2300
query60	262	248	253	248
query61	115	140	112	112
query62	703	474	463	463
query63	312	290	287	287
query64	6475	3128	3174	3128
query65	3088	3027	3028	3027
query66	1449	341	334	334
query67	15410	15329	14975	14975
query68	8686	562	583	562
query69	556	330	352	330
query70	1220	1126	1080	1080
query71	492	285	286	285
query72	6520	2760	2417	2417
query73	808	328	323	323
query74	6606	6351	6342	6342
query75	3279	2319	2340	2319
query76	5127	1022	1215	1022
query77	619	247	244	244
query78	10962	10030	10041	10030
query79	8526	538	531	531
query80	1417	428	417	417
query81	505	246	247	246
query82	708	109	110	109
query83	205	163	159	159
query84	259	87	89	87
query85	1445	301	282	282
query86	423	298	291	291
query87	3720	3508	3489	3489
query88	3622	2299	2289	2289
query89	569	376	364	364
query90	1990	177	186	177
query91	135	108	104	104
query92	60	52	53	52
query93	6891	530	519	519
query94	1182	199	192	192
query95	434	320	324	320
query96	609	276	270	270
query97	2699	2512	2498	2498
query98	246	219	207	207
query99	1268	846	831	831
Total cold run time: 296828 ms
Total hot run time: 180836 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.66% (8883/24913)
Line Coverage: 27.38% (72919/266356)
Region Coverage: 26.55% (37708/142025)
Branch Coverage: 23.35% (19219/82318)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0c3b8d301420e66b02ce00a486a5ef5814c75ce5_0c3b8d301420e66b02ce00a486a5ef5814c75ce5/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 30.63 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0c3b8d301420e66b02ce00a486a5ef5814c75ce5, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.06
query5	0.48	0.48	0.49
query6	1.15	0.66	0.65
query7	0.01	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.50	0.51
query10	0.55	0.57	0.56
query11	0.14	0.11	0.12
query12	0.14	0.11	0.12
query13	0.60	0.59	0.60
query14	0.78	0.78	0.78
query15	0.86	0.85	0.84
query16	0.36	0.36	0.36
query17	0.97	1.02	0.98
query18	0.26	0.25	0.25
query19	1.79	1.76	1.78
query20	0.01	0.01	0.01
query21	15.42	0.76	0.71
query22	3.12	5.29	2.07
query23	17.83	1.36	1.09
query24	1.32	0.34	0.22
query25	0.13	0.09	0.09
query26	0.27	0.17	0.18
query27	0.08	0.09	0.09
query28	13.69	0.95	0.94
query29	12.63	3.43	3.50
query30	0.27	0.06	0.06
query31	2.84	0.39	0.38
query32	3.30	0.46	0.46
query33	2.82	2.86	2.86
query34	15.53	4.34	4.34
query35	4.37	4.40	4.37
query36	0.66	0.48	0.48
query37	0.20	0.18	0.17
query38	0.17	0.17	0.15
query39	0.05	0.04	0.04
query40	0.18	0.16	0.14
query41	0.09	0.05	0.06
query42	0.07	0.06	0.06
query43	0.06	0.05	0.06
Total cold run time: 105.79 s
Total hot run time: 30.63 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 0c3b8d301420e66b02ce00a486a5ef5814c75ce5 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.4 seconds inserted 10000000 Rows, about 609K ops/s

@xinyiZzz xinyiZzz closed this Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants