Skip to content

[fix](merge-cloud) Fix drop rollup/mv leak meta kvs and data objects resource#33321

Merged
dataroaring merged 8 commits intoapache:masterfrom
swjtu-zhanglei:master-fix7
Apr 8, 2024
Merged

[fix](merge-cloud) Fix drop rollup/mv leak meta kvs and data objects resource#33321
dataroaring merged 8 commits intoapache:masterfrom
swjtu-zhanglei:master-fix7

Conversation

@swjtu-zhanglei
Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@swjtu-zhanglei
Copy link
Copy Markdown
Contributor Author

run buildall

1 similar comment
@swjtu-zhanglei
Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 38643 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit de5bc7567dedd32d719d2d1e24664d399675ee4f, data reload: false

------ Round 1 ----------------------------------
q1	17954	4064	4025	4025
q2	2012	185	179	179
q3	10485	1165	1300	1165
q4	10422	863	1002	863
q5	7541	2931	2951	2931
q6	219	134	133	133
q7	1102	630	599	599
q8	9414	1967	2019	1967
q9	6686	6228	6160	6160
q10	8408	3537	3501	3501
q11	406	236	237	236
q12	382	218	221	218
q13	17756	2908	2969	2908
q14	258	227	239	227
q15	538	484	467	467
q16	472	385	368	368
q17	952	920	925	920
q18	7428	6548	6617	6548
q19	1614	1524	1532	1524
q20	588	312	329	312
q21	3517	3114	3095	3095
q22	364	297	299	297
Total cold run time: 108518 ms
Total hot run time: 38643 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4023	4000	4027	4000
q2	329	217	224	217
q3	2980	2946	2951	2946
q4	1904	1899	1871	1871
q5	5272	5232	5242	5232
q6	207	125	125	125
q7	2258	1782	1772	1772
q8	3271	3273	3271	3271
q9	8555	8486	8510	8486
q10	3736	3833	3814	3814
q11	543	448	449	448
q12	695	518	536	518
q13	10698	2953	2933	2933
q14	291	254	249	249
q15	509	470	471	470
q16	451	399	401	399
q17	1717	1681	1683	1681
q18	7625	7434	7270	7270
q19	1637	1633	1633	1633
q20	1932	1718	1712	1712
q21	4972	4778	4781	4778
q22	494	396	415	396
Total cold run time: 64099 ms
Total hot run time: 54221 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 180696 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit de5bc7567dedd32d719d2d1e24664d399675ee4f, data reload: false

query1	1218	1126	1145	1126
query2	6478	1833	1774	1774
query3	6668	212	218	212
query4	23779	21376	21519	21376
query5	4239	392	396	392
query6	276	197	194	194
query7	4603	307	301	301
query8	238	208	181	181
query9	8464	2206	2187	2187
query10	578	249	261	249
query11	14919	14566	14445	14445
query12	145	88	90	88
query13	1640	371	385	371
query14	8770	6799	6744	6744
query15	207	172	185	172
query16	7166	272	267	267
query17	1005	586	564	564
query18	1905	290	289	289
query19	198	156	157	156
query20	95	91	88	88
query21	199	129	132	129
query22	4954	4811	4810	4810
query23	33570	32595	32627	32595
query24	10531	3061	3112	3061
query25	705	392	398	392
query26	1907	165	161	161
query27	2985	325	346	325
query28	6714	1814	1797	1797
query29	1390	596	589	589
query30	311	170	172	170
query31	1000	735	718	718
query32	101	55	55	55
query33	650	259	254	254
query34	901	485	505	485
query35	835	686	697	686
query36	997	905	866	866
query37	287	73	73	73
query38	3527	3387	3378	3378
query39	1577	1527	1514	1514
query40	285	129	131	129
query41	52	47	47	47
query42	114	106	101	101
query43	468	425	430	425
query44	1128	719	713	713
query45	281	265	263	263
query46	1072	799	780	780
query47	1880	1801	1781	1781
query48	376	306	302	302
query49	1173	380	373	373
query50	794	398	398	398
query51	6725	6634	6619	6619
query52	106	99	98	98
query53	369	284	293	284
query54	280	233	240	233
query55	86	78	76	76
query56	249	230	225	225
query57	1228	1107	1125	1107
query58	249	213	218	213
query59	2536	2317	2204	2204
query60	263	270	250	250
query61	116	115	114	114
query62	693	449	436	436
query63	307	285	286	285
query64	6492	3223	3271	3223
query65	3070	3005	3008	3005
query66	1444	331	310	310
query67	15549	14968	15327	14968
query68	7414	572	586	572
query69	538	314	300	300
query70	1274	1155	1117	1117
query71	504	273	267	267
query72	6355	2582	2426	2426
query73	805	328	324	324
query74	6619	6288	6342	6288
query75	3642	2256	2292	2256
query76	6021	1088	1224	1088
query77	676	249	253	249
query78	10870	10331	10127	10127
query79	8655	553	539	539
query80	1196	443	430	430
query81	515	247	245	245
query82	735	98	96	96
query83	207	179	171	171
query84	269	85	91	85
query85	1512	279	275	275
query86	459	292	281	281
query87	3722	3444	3532	3444
query88	4291	2358	2356	2356
query89	562	376	375	375
query90	2008	183	188	183
query91	138	101	106	101
query92	63	50	51	50
query93	6651	537	529	529
query94	1150	183	179	179
query95	430	318	316	316
query96	622	270	271	270
query97	2648	2497	2520	2497
query98	229	217	215	215
query99	1304	864	849	849
Total cold run time: 295201 ms
Total hot run time: 180696 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 29.9 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit de5bc7567dedd32d719d2d1e24664d399675ee4f, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.06	0.05
query4	1.68	0.07	0.06
query5	0.48	0.48	0.48
query6	1.13	0.64	0.65
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.51	0.52
query10	0.58	0.57	0.56
query11	0.16	0.11	0.12
query12	0.15	0.12	0.13
query13	0.62	0.59	0.59
query14	0.77	0.80	0.78
query15	0.87	0.85	0.84
query16	0.35	0.36	0.35
query17	0.98	1.00	1.01
query18	0.24	0.26	0.26
query19	1.81	1.71	1.72
query20	0.02	0.01	0.01
query21	15.75	0.65	0.64
query22	3.83	6.26	1.44
query23	18.09	1.43	1.33
query24	1.64	0.19	0.19
query25	0.17	0.08	0.08
query26	0.28	0.16	0.16
query27	0.08	0.07	0.08
query28	13.75	0.98	0.94
query29	12.73	3.27	3.30
query30	0.25	0.06	0.05
query31	2.85	0.39	0.41
query32	3.26	0.49	0.48
query33	2.88	2.85	2.88
query34	15.50	4.36	4.37
query35	4.40	4.38	4.38
query36	0.68	0.47	0.47
query37	0.18	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.20	0.15	0.16
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 107.72 s
Total hot run time: 29.9 s

@doris-robot
Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit de5bc7567dedd32d719d2d1e24664d399675ee4f with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.1 seconds inserted 10000000 Rows, about 621K ops/s

* @param tableId
* @param materializedIndexId
*/
public void eraseIndexDropBackendReplicas(long tableId, List<Long> indexIdList) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a cofusing name, consider eraseDroppedIndex

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a cofusing name, consider eraseDroppedIndex

resolved

@swjtu-zhanglei
Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 38898 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1bd67bb047dc57f507b87da7a14f57e2981adbe6, data reload: false

------ Round 1 ----------------------------------
q1	17622	4099	4129	4099
q2	2026	183	177	177
q3	10829	1170	1352	1170
q4	10519	868	996	868
q5	7994	3009	2998	2998
q6	225	140	147	140
q7	1028	542	536	536
q8	9682	2053	2071	2053
q9	6895	6354	6288	6288
q10	8588	3547	3511	3511
q11	421	242	237	237
q12	385	215	212	212
q13	17782	2944	2948	2944
q14	276	237	244	237
q15	527	488	472	472
q16	514	383	374	374
q17	962	897	883	883
q18	7194	6462	6449	6449
q19	1620	1541	1559	1541
q20	568	351	306	306
q21	3543	3115	3101	3101
q22	358	302	318	302
Total cold run time: 109558 ms
Total hot run time: 38898 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4086	4037	4068	4037
q2	336	223	219	219
q3	2956	2952	2945	2945
q4	1850	1888	1847	1847
q5	5239	5210	5219	5210
q6	212	127	128	127
q7	1771	1353	1350	1350
q8	3259	3311	3287	3287
q9	8508	8479	8505	8479
q10	3781	3873	3843	3843
q11	544	445	445	445
q12	702	532	552	532
q13	6001	2943	2939	2939
q14	287	255	263	255
q15	506	467	468	467
q16	444	396	384	384
q17	1704	1702	1696	1696
q18	7602	7130	7193	7130
q19	1640	1634	1639	1634
q20	1943	1715	1754	1715
q21	5065	4744	4709	4709
q22	490	440	430	430
Total cold run time: 58926 ms
Total hot run time: 53680 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 180657 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1bd67bb047dc57f507b87da7a14f57e2981adbe6, data reload: false

query1	1251	1119	1118	1118
query2	6496	1837	1780	1780
query3	6675	216	227	216
query4	25709	21521	21575	21521
query5	4203	420	410	410
query6	278	183	194	183
query7	4623	305	308	305
query8	234	185	187	185
query9	8493	2220	2225	2220
query10	589	273	258	258
query11	14940	14553	14455	14455
query12	143	94	91	91
query13	1637	384	385	384
query14	8613	6801	6800	6800
query15	212	176	185	176
query16	7161	270	267	267
query17	1022	604	565	565
query18	1909	285	293	285
query19	207	157	162	157
query20	94	90	89	89
query21	197	127	130	127
query22	4963	4754	4769	4754
query23	33299	32469	32451	32451
query24	10870	3246	3096	3096
query25	696	393	388	388
query26	1894	163	161	161
query27	3027	321	323	321
query28	6733	1824	1818	1818
query29	1376	598	597	597
query30	298	164	164	164
query31	967	720	731	720
query32	104	58	58	58
query33	650	260	253	253
query34	919	499	504	499
query35	858	709	703	703
query36	994	876	875	875
query37	289	75	75	75
query38	3477	3380	3419	3380
query39	1581	1524	1528	1524
query40	286	129	132	129
query41	48	48	46	46
query42	113	102	105	102
query43	458	409	404	404
query44	1169	727	731	727
query45	284	269	268	268
query46	1074	816	786	786
query47	1860	1763	1773	1763
query48	376	305	306	305
query49	1163	380	375	375
query50	825	396	395	395
query51	6741	6511	6616	6511
query52	111	100	98	98
query53	362	293	287	287
query54	289	245	237	237
query55	85	77	79	77
query56	246	232	238	232
query57	1216	1127	1136	1127
query58	239	220	218	218
query59	2481	2338	2299	2299
query60	262	242	253	242
query61	115	109	109	109
query62	694	450	444	444
query63	318	289	296	289
query64	6423	3167	3132	3132
query65	3106	3019	3016	3016
query66	1450	332	310	310
query67	15705	14904	14882	14882
query68	6354	574	580	574
query69	542	317	302	302
query70	1201	1114	1107	1107
query71	514	286	275	275
query72	6405	2566	2396	2396
query73	802	330	327	327
query74	6749	6325	6317	6317
query75	3548	2294	2314	2294
query76	4777	1091	1244	1091
query77	615	254	246	246
query78	10759	10187	10060	10060
query79	8492	520	532	520
query80	1541	437	422	422
query81	512	230	237	230
query82	730	100	98	98
query83	204	170	168	168
query84	265	86	89	86
query85	1498	279	281	279
query86	488	308	307	307
query87	3720	3471	3464	3464
query88	4224	2343	2339	2339
query89	576	371	374	371
query90	2025	183	189	183
query91	135	108	106	106
query92	62	47	48	47
query93	6574	557	526	526
query94	1135	181	181	181
query95	432	318	316	316
query96	602	269	270	269
query97	2664	2463	2493	2463
query98	238	222	210	210
query99	1313	844	845	844
Total cold run time: 294876 ms
Total hot run time: 180657 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 29.56 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1bd67bb047dc57f507b87da7a14f57e2981adbe6, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.08	0.08
query5	0.48	0.49	0.48
query6	1.14	0.66	0.66
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.57	0.50	0.51
query10	0.56	0.56	0.57
query11	0.14	0.11	0.12
query12	0.15	0.13	0.12
query13	0.61	0.60	0.60
query14	0.79	0.78	0.80
query15	0.86	0.84	0.86
query16	0.34	0.35	0.36
query17	0.98	0.98	0.98
query18	0.25	0.25	0.26
query19	1.80	1.75	1.73
query20	0.01	0.01	0.01
query21	15.43	0.64	0.65
query22	4.03	7.75	1.11
query23	18.12	1.28	1.30
query24	1.51	0.21	0.20
query25	0.15	0.07	0.08
query26	0.27	0.17	0.17
query27	0.08	0.09	0.07
query28	13.86	0.97	0.94
query29	12.55	3.28	3.29
query30	0.25	0.06	0.06
query31	2.88	0.40	0.40
query32	3.25	0.49	0.48
query33	2.84	2.90	2.92
query34	15.49	4.33	4.33
query35	4.40	4.40	4.36
query36	0.67	0.47	0.48
query37	0.18	0.16	0.15
query38	0.16	0.13	0.14
query39	0.04	0.03	0.04
query40	0.20	0.14	0.16
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.32 s
Total hot run time: 29.56 s

@doris-robot
Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1bd67bb047dc57f507b87da7a14f57e2981adbe6 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.1 seconds inserted 10000000 Rows, about 621K ops/s

dataroaring and others added 7 commits April 7, 2024 23:30
…udInternalCatalog.java

Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
…ewHandler.java

Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
…ewHandler.java

Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
…udInternalCatalog.java

Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
…ewHandler.java

Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
…udInternalCatalog.java

Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
…atalog.java

Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>
@dataroaring
Copy link
Copy Markdown
Contributor

run buildall

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 7, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 38724 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c5d8ed58cd05c78d9a1bb072ee319e3eb74f0cb, data reload: false

------ Round 1 ----------------------------------
q1	17608	4251	4236	4236
q2	1994	176	162	162
q3	10526	1191	1338	1191
q4	10196	806	982	806
q5	7495	2922	2924	2922
q6	220	137	134	134
q7	1107	652	615	615
q8	9414	2064	2046	2046
q9	6760	6248	6144	6144
q10	8459	3510	3522	3510
q11	422	234	250	234
q12	376	212	209	209
q13	17784	2878	2904	2878
q14	283	237	237	237
q15	512	484	472	472
q16	492	378	376	376
q17	966	889	893	889
q18	7375	6493	6381	6381
q19	1613	1568	1554	1554
q20	561	307	295	295
q21	3516	3153	3125	3125
q22	353	309	308	308
Total cold run time: 108032 ms
Total hot run time: 38724 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4081	4063	4043	4043
q2	337	232	219	219
q3	2947	2958	2958	2958
q4	1861	1822	1807	1807
q5	5224	5203	5196	5196
q6	215	125	125	125
q7	2209	1789	1796	1789
q8	3229	3276	3308	3276
q9	8456	8469	8450	8450
q10	3744	3830	3841	3830
q11	551	460	434	434
q12	716	537	579	537
q13	16729	2891	2885	2885
q14	286	262	257	257
q15	514	481	466	466
q16	444	399	401	399
q17	1724	1684	1671	1671
q18	7666	7717	7590	7590
q19	1750	1702	1716	1702
q20	2072	1871	1868	1868
q21	5290	5032	4939	4939
q22	513	462	461	461
Total cold run time: 70558 ms
Total hot run time: 54902 ms

Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 7, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 7, 2024

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 6dbf349 into apache:master Apr 8, 2024
seawinde pushed a commit to seawinde/doris that referenced this pull request Apr 10, 2024
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Nov 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants