Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve](restore) Release useless info for the finished job to reduce mem usage #34556

Merged
merged 1 commit into from
May 8, 2024

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented May 8, 2024

Cherry-pick #34513

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented May 8, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49740 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7145ca40c5b73573d4e29696b86a5b98c88d8272, data reload: false

------ Round 1 ----------------------------------
q1	17646	4358	4339	4339
q2	2058	152	145	145
q3	10528	1898	1892	1892
q4	10330	1286	1302	1286
q5	8413	3923	3923	3923
q6	225	125	125	125
q7	2033	1582	1586	1582
q8	9308	2701	2696	2696
q9	10967	10522	10417	10417
q10	8668	3512	3481	3481
q11	419	239	244	239
q12	456	299	302	299
q13	18383	3919	4053	3919
q14	356	331	331	331
q15	504	462	454	454
q16	665	565	562	562
q17	1110	941	952	941
q18	7295	6877	7021	6877
q19	1709	1541	1559	1541
q20	537	329	307	307
q21	4466	4052	4001	4001
q22	491	389	383	383
Total cold run time: 116567 ms
Total hot run time: 49740 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4285	4297	4275	4275
q2	322	217	228	217
q3	4165	4124	4145	4124
q4	2741	2739	2732	2732
q5	7185	7115	7076	7076
q6	234	123	117	117
q7	3237	2803	2794	2794
q8	4348	4423	4406	4406
q9	17317	16940	16865	16865
q10	4240	4217	4219	4217
q11	729	710	701	701
q12	1016	857	851	851
q13	7181	3771	3727	3727
q14	448	428	417	417
q15	505	460	457	457
q16	748	702	666	666
q17	3733	3831	3900	3831
q18	8766	8783	8794	8783
q19	1703	1690	1644	1644
q20	2420	2144	2084	2084
q21	8434	8376	8371	8371
q22	1022	923	913	913
Total cold run time: 84779 ms
Total hot run time: 79268 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203416 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7145ca40c5b73573d4e29696b86a5b98c88d8272, data reload: false

query1	929	385	380	380
query2	6525	2731	2648	2648
query3	6921	211	200	200
query4	20594	18012	17991	17991
query5	19722	6524	6552	6524
query6	318	212	222	212
query7	4534	299	305	299
query8	262	241	233	233
query9	3195	2729	2665	2665
query10	431	290	309	290
query11	11356	10704	10714	10704
query12	120	72	76	72
query13	5605	683	699	683
query14	18419	13563	13668	13563
query15	356	217	219	217
query16	6443	264	262	262
query17	1472	1506	855	855
query18	2285	404	408	404
query19	212	151	149	149
query20	78	76	76	76
query21	182	98	94	94
query22	5248	5159	5118	5118
query23	32587	31872	31936	31872
query24	6898	6514	6455	6455
query25	533	435	435	435
query26	510	172	163	163
query27	1773	300	308	300
query28	6167	2383	2360	2360
query29	2901	2717	2618	2618
query30	243	165	159	159
query31	912	731	717	717
query32	68	62	60	60
query33	408	252	252	252
query34	849	463	504	463
query35	1141	935	910	910
query36	1327	1034	1241	1034
query37	93	61	63	61
query38	3096	2919	2912	2912
query39	1364	1331	1315	1315
query40	203	98	99	98
query41	38	34	34	34
query42	82	85	88	85
query43	783	758	671	671
query44	1150	723	716	716
query45	242	225	229	225
query46	1222	987	967	967
query47	2055	1771	1696	1696
query48	999	706	711	706
query49	613	382	390	382
query50	876	611	625	611
query51	4816	4669	4711	4669
query52	89	84	81	81
query53	445	318	322	318
query54	2670	2505	2481	2481
query55	87	79	82	79
query56	218	198	211	198
query57	1137	1132	1079	1079
query58	220	197	189	189
query59	4314	3943	3928	3928
query60	213	205	210	205
query61	90	83	83	83
query62	795	492	477	477
query63	475	346	341	341
query64	2471	1519	1544	1519
query65	3634	3549	3562	3549
query66	759	369	390	369
query67	16051	15283	15205	15205
query68	8539	649	664	649
query69	581	350	351	350
query70	1610	1466	1613	1466
query71	395	297	313	297
query72	6488	3502	3425	3425
query73	733	320	329	320
query74	6340	5854	5833	5833
query75	5107	3750	3758	3750
query76	5191	1139	1204	1139
query77	854	246	269	246
query78	12848	11568	11636	11568
query79	8446	638	646	638
query80	1398	393	394	393
query81	485	234	227	227
query82	1698	107	100	100
query83	166	134	138	134
query84	252	68	70	68
query85	849	307	300	300
query86	339	291	281	281
query87	3204	3051	3002	3002
query88	5047	2384	2394	2384
query89	379	298	295	295
query90	1955	207	216	207
query91	166	132	133	132
query92	57	50	53	50
query93	5870	568	542	542
query94	727	211	204	204
query95	1102	1069	1054	1054
query96	650	342	334	334
query97	6420	6307	6375	6307
query98	177	181	173	173
query99	2966	958	862	862
Total cold run time: 313863 ms
Total hot run time: 203416 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.37 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7145ca40c5b73573d4e29696b86a5b98c88d8272, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.03
query3	0.24	0.04	0.04
query4	1.81	0.07	0.06
query5	0.53	0.52	0.52
query6	1.40	0.62	0.62
query7	0.01	0.01	0.01
query8	0.04	0.02	0.02
query9	0.52	0.48	0.47
query10	0.55	0.55	0.53
query11	0.11	0.09	0.09
query12	0.12	0.09	0.09
query13	0.61	0.61	0.60
query14	0.78	0.80	0.78
query15	0.78	0.75	0.76
query16	0.35	0.37	0.36
query17	1.00	1.01	0.98
query18	0.25	0.24	0.24
query19	1.95	1.82	1.85
query20	0.01	0.01	0.01
query21	15.85	0.55	0.53
query22	2.24	2.08	2.32
query23	17.49	1.05	0.95
query24	4.98	3.94	1.12
query25	0.40	0.06	0.05
query26	0.83	0.15	0.15
query27	0.05	0.04	0.03
query28	5.32	0.77	0.74
query29	12.68	2.36	2.36
query30	0.60	0.55	0.55
query31	2.81	0.37	0.37
query32	3.41	0.48	0.51
query33	3.08	3.02	3.10
query34	15.27	4.81	4.81
query35	4.87	4.88	4.84
query36	1.08	1.00	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.15	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 102.5 s
Total hot run time: 31.37 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7145ca40c5b73573d4e29696b86a5b98c88d8272 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@dataroaring dataroaring merged commit 7268125 into apache:branch-2.0 May 8, 2024
24 of 26 checks passed
@w41ter w41ter deleted the pick/34513 branch May 9, 2024 01:37
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants