Skip to content

[fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. #31212#31233

Merged
xiaokang merged 1 commit intoapache:branch-2.0from
Jibing-Li:leak2.0
Feb 21, 2024
Merged

[fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. #31212#31233
xiaokang merged 1 commit intoapache:branch-2.0from
Jibing-Li:leak2.0

Conversation

@Jibing-Li
Copy link
Contributor

backport #31212

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@Jibing-Li Jibing-Li marked this pull request as ready for review February 21, 2024 11:40
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50119 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02493752253d23747e1d36a226d30024b3544561, data reload: false

------ Round 1 ----------------------------------
q1	20211	4547	4539	4539
q2	2698	159	154	154
q3	11403	1963	1945	1945
q4	10102	1241	1316	1241
q5	8394	3935	3969	3935
q6	233	125	122	122
q7	1996	1596	1620	1596
q8	9255	2687	2702	2687
q9	10843	10485	10454	10454
q10	8600	3531	3498	3498
q11	423	247	248	247
q12	462	290	297	290
q13	18342	3993	4062	3993
q14	342	326	342	326
q15	516	455	449	449
q16	698	586	584	584
q17	1119	979	965	965
q18	7191	6928	6808	6808
q19	1702	1525	1522	1522
q20	529	302	308	302
q21	4424	4099	4071	4071
q22	519	403	391	391
Total cold run time: 120002 ms
Total hot run time: 50119 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4318	4433	4329	4329
q2	315	228	231	228
q3	4157	4154	4131	4131
q4	2749	2743	2750	2743
q5	7282	7231	7238	7231
q6	231	119	117	117
q7	3217	2800	2807	2800
q8	4387	4467	4487	4467
q9	17089	17081	17126	17081
q10	4233	4294	4282	4282
q11	745	665	674	665
q12	1009	852	866	852
q13	6927	3752	3760	3752
q14	445	426	421	421
q15	491	456	472	456
q16	756	712	692	692
q17	3794	3835	3855	3835
q18	8719	8740	8842	8740
q19	1701	1698	1638	1638
q20	2450	2124	2118	2118
q21	8401	8346	8328	8328
q22	1062	958	932	932
Total cold run time: 84478 ms
Total hot run time: 79838 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 243031 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 02493752253d23747e1d36a226d30024b3544561, data reload: false

query1	934	396	378	378
query2	6518	2348	2110	2110
query3	6920	198	198	198
query4	20690	17967	18058	17967
query5	19741	6530	6505	6505
query6	293	213	216	213
query7	4155	290	289	289
query8	244	242	224	224
query9	3119	2701	2631	2631
query10	395	294	307	294
query11	11356	10646	10607	10607
query12	115	72	72	72
query13	5571	646	625	625
query14	17594	13142	13705	13142
query15	356	240	245	240
query16	6453	260	260	260
query17	1734	1450	865	865
query18	2329	408	415	408
query19	205	153	147	147
query20	71	75	77	75
query21	186	89	97	89
query22	5275	5130	5065	5065
query23	32428	31903	32004	31903
query24	7719	6477	6509	6477
query25	537	428	437	428
query26	1019	159	155	155
query27	2208	293	290	290
query28	6041	2206	2153	2153
query29	2799	2854	2717	2717
query30	237	164	162	162
query31	918	726	742	726
query32	67	59	61	59
query33	396	255	268	255
query34	848	465	478	465
query35	1101	895	876	876
query36	1603	1715	1389	1389
query37	94	60	60	60
query38	3090	2885	2895	2885
query39	1369	1315	1318	1315
query40	295	98	94	94
query41	34	32	30	30
query42	82	75	83	75
query43	614	638	593	593
query44	1151	697	723	697
query45	237	232	225	225
query46	1233	966	960	960
query47	1843	1708	1908	1708
query48	968	673	687	673
query49	637	357	362	357
query50	863	605	571	571
query51	4767	4586	4610	4586
query52	103	73	80	73
query53	447	322	316	316
query54	2669	2446	2510	2446
query55	89	74	71	71
query56	195	200	208	200
query57	1286	1106	1144	1106
query58	210	207	196	196
query59	3600	3244	3272	3244
query60	213	190	206	190
query61	82	80	79	79
query62	847	457	481	457
query63	474	327	336	327
query64	5148	1519	1367	1367
query65	3643	3569	3552	3552
query66	655	365	365	365
query67	16006	17812	15188	15188
query68	8384	619	667	619
query69	563	348	340	340
query70	1944	1573	1610	1573
query71	390	313	308	308
query72	6158	3430	3478	3430
query73	732	317	319	317
query74	6306	5828	5797	5797
query75	5104	3731	3639	3639
query76	5029	1141	1233	1141
query77	742	250	251	250
query78	31658	53542	55657	53542
query79	15805	634	619	619
query80	5055	384	371	371
query81	570	227	228	227
query82	2002	97	90	90
query83	331	125	134	125
query84	259	67	66	66
query85	1973	283	282	282
query86	461	376	359	359
query87	3248	2980	3016	2980
query88	7267	2328	2304	2304
query89	478	289	279	279
query90	2530	204	209	204
query91	165	124	125	124
query92	62	50	53	50
query93	6608	554	596	554
query94	1831	204	207	204
query95	1095	1061	1052	1052
query96	666	329	326	326
query97	6446	6330	6370	6330
query98	194	172	168	168
query99	3778	896	928	896
Total cold run time: 352953 ms
Total hot run time: 243031 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02493752253d23747e1d36a226d30024b3544561, data reload: false

query1	0.03	0.02	0.02
query2	0.06	0.02	0.02
query3	0.25	0.05	0.05
query4	1.81	0.06	0.07
query5	0.54	0.52	0.54
query6	1.23	0.62	0.67
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.49	0.48
query10	0.53	0.54	0.53
query11	0.11	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.61	0.60
query14	0.77	0.78	0.80
query15	0.78	0.76	0.78
query16	0.37	0.38	0.38
query17	0.99	1.01	0.96
query18	0.21	0.26	0.26
query19	1.92	1.73	1.83
query20	0.02	0.01	0.02
query21	15.47	0.57	0.55
query22	1.98	2.07	1.43
query23	17.04	0.89	1.10
query24	5.49	3.23	0.91
query25	1.48	0.12	0.11
query26	0.35	0.15	0.13
query27	0.11	0.11	0.11
query28	6.50	0.74	0.71
query29	12.68	2.26	2.32
query30	0.56	0.49	0.51
query31	2.83	0.38	0.39
query32	3.39	0.49	0.49
query33	3.06	3.07	3.05
query34	15.25	4.78	4.79
query35	4.83	4.85	4.82
query36	1.05	1.02	1.01
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.15	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 103.38 s
Total hot run time: 30.36 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 02493752253d23747e1d36a226d30024b3544561 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@xiaokang xiaokang merged commit 1910f29 into apache:branch-2.0 Feb 21, 2024
@Jibing-Li Jibing-Li deleted the leak2.0 branch February 21, 2024 14:29
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants