[branch-2.0](tablet invert index) fix tablet invert index leaky caused by auto partition #33973

[yujun777]

pick: #33973

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[yujun777]

run buildall

[dataroaring]

auto partition does not exist in 2.0.

[yujun777]

not only auto partition will cause concurrent partitions with the same name. The create table at first time and dynamic partition sched thread will also cause too.

[yujun777]

run buildall

[yujun777]

run buildall

[doris-robot]

TPC-H: Total hot run time: 49928 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca987044ce108867f6da81c7588e967999a5bc5b, data reload: false

------ Round 1 ----------------------------------
q1	17759	4445	4401	4401
q2	2025	149	141	141
q3	10308	1927	1929	1927
q4	10100	1263	1361	1263
q5	8584	3859	3932	3859
q6	229	121	123	121
q7	2015	1598	1580	1580
q8	9325	2715	2723	2715
q9	11036	10658	10539	10539
q10	8626	3481	3526	3481
q11	422	243	240	240
q12	462	291	299	291
q13	18343	3976	4031	3976
q14	353	319	322	319
q15	507	456	459	456
q16	677	576	568	568
q17	1125	946	947	946
q18	7288	6789	6899	6789
q19	1684	1544	1520	1520
q20	536	302	316	302
q21	4416	4108	4141	4108
q22	499	386	395	386
Total cold run time: 116319 ms
Total hot run time: 49928 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4333	4310	4287	4287
q2	314	219	223	219
q3	4181	4111	4120	4111
q4	2734	2737	2727	2727
q5	7159	7172	7129	7129
q6	235	119	118	118
q7	3215	2863	2850	2850
q8	4361	4442	4447	4442
q9	17331	17068	17091	17068
q10	4206	4251	4246	4246
q11	750	685	685	685
q12	1011	834	870	834
q13	6783	3738	3721	3721
q14	445	427	432	427
q15	493	455	462	455
q16	733	670	683	670
q17	3811	3802	3850	3802
q18	8839	8858	8720	8720
q19	1698	1708	1640	1640
q20	2391	2168	2139	2139
q21	8433	8433	8437	8433
q22	1001	975	936	936
Total cold run time: 84457 ms
Total hot run time: 79659 ms

[doris-robot]

TPC-DS: Total hot run time: 203677 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca987044ce108867f6da81c7588e967999a5bc5b, data reload: false

query1	914	389	378	378
query2	6524	2850	2642	2642
query3	6918	209	204	204
query4	19825	17879	17843	17843
query5	19716	6580	6498	6498
query6	276	213	230	213
query7	4146	292	297	292
query8	275	266	254	254
query9	3138	2692	2614	2614
query10	426	283	306	283
query11	11307	10648	10721	10648
query12	120	76	69	69
query13	5607	677	676	676
query14	17611	13292	14065	13292
query15	376	220	230	220
query16	6467	271	254	254
query17	1777	1461	884	884
query18	2288	407	397	397
query19	214	159	146	146
query20	77	77	76	76
query21	191	92	96	92
query22	5234	4962	4982	4962
query23	32554	31911	31909	31909
query24	6965	6532	6477	6477
query25	516	451	416	416
query26	532	170	163	163
query27	1838	300	297	297
query28	6132	2380	2324	2324
query29	2764	2707	2779	2707
query30	237	161	160	160
query31	900	746	708	708
query32	69	60	57	57
query33	390	252	237	237
query34	844	467	476	467
query35	1156	945	931	931
query36	1427	1168	1273	1168
query37	91	63	58	58
query38	3061	2898	2896	2896
query39	1371	1311	1300	1300
query40	198	95	101	95
query41	37	36	34	34
query42	83	85	82	82
query43	815	658	645	645
query44	1098	711	712	711
query45	242	228	230	228
query46	1224	973	952	952
query47	1786	1831	1689	1689
query48	999	711	705	705
query49	609	370	377	370
query50	864	614	616	614
query51	4760	4637	4566	4566
query52	95	78	75	75
query53	444	317	311	311
query54	2639	2476	2482	2476
query55	88	82	80	80
query56	227	209	208	208
query57	1197	1110	1181	1110
query58	218	198	197	197
query59	3992	4135	4270	4135
query60	207	208	209	208
query61	90	83	89	83
query62	809	496	465	465
query63	479	336	333	333
query64	2551	1518	1461	1461
query65	3669	3566	3529	3529
query66	764	380	382	380
query67	16646	15470	15742	15470
query68	10575	650	672	650
query69	572	322	353	322
query70	1996	1411	1468	1411
query71	426	303	317	303
query72	6508	3494	3459	3459
query73	726	326	321	321
query74	6277	5810	5879	5810
query75	5364	3693	3655	3655
query76	6766	1135	1161	1135
query77	1184	276	249	249
query78	12840	13159	12147	12147
query79	7445	635	644	635
query80	665	398	413	398
query81	489	227	230	227
query82	488	99	99	99
query83	165	137	134	134
query84	261	70	70	70
query85	693	310	303	303
query86	343	287	299	287
query87	3253	3018	3076	3018
query88	3960	2363	2357	2357
query89	343	298	275	275
query90	1895	219	214	214
query91	175	132	145	132
query92	55	56	50	50
query93	3635	553	569	553
query94	665	205	204	204
query95	1099	1077	1053	1053
query96	644	334	330	330
query97	6527	6408	6436	6408
query98	185	185	164	164
query99	2740	894	880	880
Total cold run time: 310063 ms
Total hot run time: 203677 ms

[doris-robot]

ClickBench: Total hot run time: 30.61 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ca987044ce108867f6da81c7588e967999a5bc5b, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.05
query4	1.79	0.08	0.07
query5	0.53	0.52	0.53
query6	1.28	0.61	0.61
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.49	0.48
query10	0.54	0.52	0.53
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.61	0.62	0.61
query14	0.79	0.78	0.77
query15	0.78	0.76	0.77
query16	0.37	0.37	0.36
query17	1.01	0.99	0.99
query18	0.21	0.26	0.24
query19	1.92	1.78	1.82
query20	0.02	0.01	0.01
query21	15.47	0.54	0.56
query22	1.99	2.01	1.60
query23	17.17	0.97	1.08
query24	7.56	1.06	0.89
query25	0.37	0.11	0.06
query26	0.82	0.16	0.15
query27	0.05	0.04	0.03
query28	5.70	0.77	0.73
query29	12.62	2.30	2.25
query30	0.57	0.53	0.51
query31	2.83	0.38	0.38
query32	3.41	0.50	0.50
query33	3.08	3.08	3.07
query34	15.25	4.81	4.79
query35	4.89	4.86	4.89
query36	1.04	1.03	1.01
query37	0.06	0.05	0.05
query38	0.04	0.02	0.03
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 104.2 s
Total hot run time: 30.61 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ca987044ce108867f6da81c7588e967999a5bc5b with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

[yujun777]

run buildall

[doris-robot]

TPC-H: Total hot run time: 49809 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ddac6ebcc8fe97531401228eec87fbea2cf5b1a0, data reload: false

------ Round 1 ----------------------------------
q1	17699	4392	4332	4332
q2	2035	158	142	142
q3	10383	1922	1870	1870
q4	10374	1225	1298	1225
q5	8402	3907	3878	3878
q6	221	126	125	125
q7	2019	1616	1586	1586
q8	9303	2698	2697	2697
q9	10854	10656	10427	10427
q10	8637	3541	3502	3502
q11	420	246	252	246
q12	460	303	301	301
q13	19115	4019	4027	4019
q14	359	328	320	320
q15	522	458	452	452
q16	685	583	562	562
q17	1114	963	979	963
q18	7248	6911	6781	6781
q19	1698	1583	1585	1583
q20	519	313	308	308
q21	4511	4085	4099	4085
q22	509	405	405	405
Total cold run time: 117087 ms
Total hot run time: 49809 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4512	4496	4449	4449
q2	341	248	244	244
q3	4289	4274	4221	4221
q4	2827	2811	2843	2811
q5	7515	7290	7168	7168
q6	235	117	117	117
q7	3235	2796	2800	2796
q8	4391	4479	4432	4432
q9	17431	17138	16660	16660
q10	4252	4218	4271	4218
q11	746	687	697	687
q12	1019	849	866	849
q13	7325	3748	3748	3748
q14	443	424	418	418
q15	503	457	451	451
q16	741	693	664	664
q17	3821	3807	3813	3807
q18	8770	8666	9381	8666
q19	1697	1704	1647	1647
q20	2421	2179	2107	2107
q21	8454	8471	8381	8381
q22	1008	979	908	908
Total cold run time: 85976 ms
Total hot run time: 79449 ms

[doris-robot]

TPC-DS: Total hot run time: 203177 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ddac6ebcc8fe97531401228eec87fbea2cf5b1a0, data reload: false

query1	917	390	376	376
query2	6545	2760	2734	2734
query3	6922	203	204	203
query4	20024	18034	18122	18034
query5	19712	6520	6524	6520
query6	283	215	215	215
query7	4151	300	296	296
query8	260	238	257	238
query9	3075	2676	2600	2600
query10	428	293	294	293
query11	11289	10691	10712	10691
query12	121	75	66	66
query13	5573	688	680	680
query14	17725	13263	13853	13263
query15	357	222	223	222
query16	6444	267	259	259
query17	1721	1460	887	887
query18	2298	406	401	401
query19	199	143	144	143
query20	71	74	76	74
query21	188	96	93	93
query22	5122	5179	5040	5040
query23	32451	31992	31885	31885
query24	6909	6520	6459	6459
query25	537	410	408	408
query26	525	166	156	156
query27	1920	287	293	287
query28	6190	2351	2320	2320
query29	2861	2586	2860	2586
query30	238	162	158	158
query31	916	724	732	724
query32	65	60	60	60
query33	397	246	243	243
query34	863	462	467	462
query35	1116	968	917	917
query36	1210	1234	1331	1234
query37	91	61	58	58
query38	3071	2974	2876	2876
query39	1361	1332	1309	1309
query40	199	89	98	89
query41	37	37	34	34
query42	83	86	82	82
query43	738	646	697	646
query44	1118	708	699	699
query45	243	229	226	226
query46	1218	967	974	967
query47	2066	1641	1796	1641
query48	1007	709	696	696
query49	619	375	376	375
query50	873	634	650	634
query51	4783	4646	4631	4631
query52	83	88	82	82
query53	454	313	327	313
query54	2671	2438	2470	2438
query55	91	82	88	82
query56	228	207	210	207
query57	1242	1052	1100	1052
query58	219	187	198	187
query59	4197	3907	3867	3867
query60	215	202	201	201
query61	91	84	86	84
query62	829	471	452	452
query63	469	328	339	328
query64	2549	1623	1379	1379
query65	3613	3559	3571	3559
query66	754	366	394	366
query67	16277	15425	15805	15425
query68	8882	649	658	649
query69	568	332	371	332
query70	1730	1405	1415	1405
query71	413	302	303	302
query72	6569	3456	3473	3456
query73	741	318	327	318
query74	6315	5945	5916	5916
query75	5441	3669	3712	3669
query76	5432	1180	1224	1180
query77	927	250	248	248
query78	12992	11755	11950	11755
query79	7361	637	641	637
query80	981	405	397	397
query81	482	228	224	224
query82	1505	97	96	96
query83	178	135	136	135
query84	256	67	69	67
query85	867	307	298	298
query86	336	306	303	303
query87	3215	3026	2974	2974
query88	5002	2360	2371	2360
query89	370	279	304	279
query90	1929	200	211	200
query91	165	134	132	132
query92	55	49	54	49
query93	4205	594	541	541
query94	698	205	205	205
query95	1081	1034	1035	1034
query96	653	335	333	333
query97	6498	6426	6492	6426
query98	184	180	169	169
query99	2914	876	928	876
Total cold run time: 310057 ms
Total hot run time: 203177 ms

[doris-robot]

ClickBench: Total hot run time: 30.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ddac6ebcc8fe97531401228eec87fbea2cf5b1a0, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.24	0.05	0.05
query4	1.80	0.08	0.06
query5	0.52	0.53	0.52
query6	1.29	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.03	0.03
query9	0.52	0.49	0.49
query10	0.55	0.53	0.54
query11	0.12	0.08	0.08
query12	0.11	0.09	0.10
query13	0.61	0.62	0.61
query14	0.79	0.77	0.79
query15	0.81	0.76	0.77
query16	0.37	0.36	0.37
query17	1.01	1.00	1.01
query18	0.20	0.26	0.23
query19	1.96	1.86	1.86
query20	0.02	0.01	0.00
query21	15.48	0.54	0.55
query22	2.31	2.83	1.37
query23	17.18	1.01	0.82
query24	5.67	0.85	1.46
query25	0.40	0.06	0.06
query26	0.65	0.15	0.15
query27	0.05	0.03	0.04
query28	7.16	0.77	0.70
query29	12.65	2.31	2.29
query30	0.54	0.48	0.51
query31	2.81	0.40	0.37
query32	3.38	0.50	0.50
query33	3.08	3.06	3.09
query34	15.26	4.79	4.79
query35	4.87	4.87	4.85
query36	1.03	1.02	1.01
query37	0.05	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 103.93 s
Total hot run time: 30.22 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ddac6ebcc8fe97531401228eec87fbea2cf5b1a0 with default session variables
Stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s