Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](build index)Remove index_meta in tablet schema when the index is dropped. #37646

Merged
merged 3 commits into from
Jul 15, 2024

Conversation

qidaye
Copy link
Contributor

@qidaye qidaye commented Jul 11, 2024

Proposed changes

Issue Number: close #37645

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Jul 11, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39977 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1fcc3a364db697d5cb28fce8f696c7fb680e84bb, data reload: false

------ Round 1 ----------------------------------
q1	18231	4432	4340	4340
q2	2361	228	212	212
q3	11277	1225	1115	1115
q4	10658	726	984	726
q5	7651	2663	2647	2647
q6	222	138	137	137
q7	971	605	593	593
q8	9258	2107	2065	2065
q9	8719	6530	6614	6530
q10	8610	3775	3722	3722
q11	460	240	238	238
q12	398	235	229	229
q13	17887	2970	2951	2951
q14	281	235	239	235
q15	529	484	476	476
q16	485	385	379	379
q17	961	712	731	712
q18	7957	7631	7434	7434
q19	4663	1509	1448	1448
q20	703	326	322	322
q21	4817	3180	3130	3130
q22	386	340	336	336
Total cold run time: 117485 ms
Total hot run time: 39977 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4368	4222	4263	4222
q2	376	252	251	251
q3	3018	2743	2763	2743
q4	1862	1595	1599	1595
q5	5263	5284	5287	5284
q6	216	128	128	128
q7	2104	1718	1766	1718
q8	3179	3320	3297	3297
q9	8410	8299	8315	8299
q10	3897	3613	3652	3613
q11	578	508	492	492
q12	807	615	627	615
q13	17611	2937	2966	2937
q14	293	270	267	267
q15	511	487	474	474
q16	479	411	415	411
q17	1743	1471	1463	1463
q18	7519	7589	7255	7255
q19	1719	1605	1638	1605
q20	1995	1790	1775	1775
q21	4884	4686	4780	4686
q22	602	542	528	528
Total cold run time: 71434 ms
Total hot run time: 53658 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173870 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1fcc3a364db697d5cb28fce8f696c7fb680e84bb, data reload: false

query1	918	378	364	364
query2	6462	2471	2446	2446
query3	6652	214	220	214
query4	28820	17577	17341	17341
query5	4178	466	460	460
query6	293	182	169	169
query7	4599	291	287	287
query8	318	298	291	291
query9	8550	2413	2408	2408
query10	433	274	262	262
query11	10724	10158	10117	10117
query12	138	83	83	83
query13	1647	370	373	370
query14	10265	7617	7561	7561
query15	241	183	191	183
query16	7904	303	305	303
query17	1834	553	520	520
query18	1987	271	272	271
query19	188	147	146	146
query20	89	79	78	78
query21	210	129	125	125
query22	4388	4161	4281	4161
query23	33701	32990	33270	32990
query24	12138	2864	2888	2864
query25	649	361	364	361
query26	1819	146	144	144
query27	3052	270	264	264
query28	7716	2099	2083	2083
query29	1150	621	605	605
query30	288	151	149	149
query31	999	754	755	754
query32	100	53	54	53
query33	768	294	282	282
query34	1032	485	496	485
query35	659	573	565	565
query36	1092	974	951	951
query37	288	75	79	75
query38	2838	2726	2805	2726
query39	862	792	794	792
query40	278	120	118	118
query41	53	54	55	54
query42	115	98	105	98
query43	590	565	538	538
query44	1223	740	739	739
query45	198	165	165	165
query46	1081	726	707	707
query47	1901	1799	1809	1799
query48	376	298	295	295
query49	1184	413	419	413
query50	789	399	398	398
query51	6935	6683	6821	6683
query52	108	90	99	90
query53	369	295	286	286
query54	1027	490	447	447
query55	79	72	74	72
query56	281	261	284	261
query57	1122	1079	1079	1079
query58	266	244	241	241
query59	3495	3367	3434	3367
query60	299	267	272	267
query61	95	91	94	91
query62	837	651	670	651
query63	326	292	285	285
query64	10390	2217	1669	1669
query65	3164	3079	3091	3079
query66	1366	339	326	326
query67	15392	15309	15037	15037
query68	4580	536	552	536
query69	472	331	317	317
query70	1173	1062	1150	1062
query71	420	279	287	279
query72	7042	5695	5334	5334
query73	750	321	326	321
query74	6075	5507	5522	5507
query75	3387	2665	2659	2659
query76	2789	962	906	906
query77	573	300	297	297
query78	9593	9002	8900	8900
query79	2274	520	525	520
query80	1514	462	481	462
query81	583	221	219	219
query82	810	132	130	130
query83	239	173	169	169
query84	239	88	91	88
query85	2079	309	300	300
query86	482	327	307	307
query87	3289	3120	3146	3120
query88	3820	2364	2387	2364
query89	475	375	380	375
query90	1862	192	187	187
query91	130	105	102	102
query92	63	50	49	49
query93	2438	502	512	502
query94	1273	279	212	212
query95	403	313	308	308
query96	611	276	267	267
query97	3198	3049	3038	3038
query98	224	200	203	200
query99	1699	1275	1292	1275
Total cold run time: 287887 ms
Total hot run time: 173870 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.98 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1fcc3a364db697d5cb28fce8f696c7fb680e84bb, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.09	0.09
query5	0.51	0.49	0.48
query6	1.14	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.05	0.04
query9	0.55	0.48	0.48
query10	0.55	0.56	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.13
query13	0.60	0.60	0.58
query14	0.75	0.77	0.77
query15	0.84	0.80	0.80
query16	0.36	0.38	0.35
query17	0.97	1.02	1.03
query18	0.23	0.21	0.21
query19	1.82	1.66	1.70
query20	0.01	0.01	0.01
query21	15.42	0.78	0.67
query22	3.73	8.17	1.37
query23	18.29	1.37	1.23
query24	2.15	0.23	0.22
query25	0.15	0.08	0.09
query26	0.30	0.21	0.21
query27	0.46	0.23	0.23
query28	13.22	1.02	0.99
query29	12.60	3.30	3.31
query30	0.25	0.06	0.04
query31	2.88	0.38	0.40
query32	3.30	0.48	0.47
query33	2.84	2.94	2.90
query34	17.19	4.36	4.30
query35	4.43	4.42	4.41
query36	0.65	0.48	0.48
query37	0.19	0.15	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.13	0.11
query41	0.08	0.04	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.31 s
Total hot run time: 29.98 s

airborne12
airborne12 previously approved these changes Jul 11, 2024
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 11, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@qidaye
Copy link
Contributor Author

qidaye commented Jul 11, 2024

run p0

1 similar comment
@qidaye
Copy link
Contributor Author

qidaye commented Jul 11, 2024

run p0

@qidaye
Copy link
Contributor Author

qidaye commented Jul 12, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 12, 2024
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39812 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2c7c710a69301e079265c70bc18b48d37f3babaa, data reload: false

------ Round 1 ----------------------------------
q1	17983	4466	4382	4382
q2	2958	195	187	187
q3	11757	1171	1105	1105
q4	11431	848	795	795
q5	7792	2702	2608	2608
q6	221	135	137	135
q7	959	602	597	597
q8	9222	2064	2090	2064
q9	8704	6529	6569	6529
q10	8696	3792	3748	3748
q11	453	234	242	234
q12	392	223	234	223
q13	17766	2993	3014	2993
q14	268	233	245	233
q15	515	474	479	474
q16	487	389	375	375
q17	965	703	697	697
q18	8064	7415	7273	7273
q19	7642	1399	1368	1368
q20	713	330	327	327
q21	4948	3243	3178	3178
q22	356	287	292	287
Total cold run time: 122292 ms
Total hot run time: 39812 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4448	4273	4295	4273
q2	383	269	275	269
q3	2998	2769	2765	2765
q4	1888	1605	1560	1560
q5	5270	5312	5296	5296
q6	222	131	134	131
q7	2150	1746	1723	1723
q8	3221	3337	3313	3313
q9	8453	8346	8436	8346
q10	3915	3703	3709	3703
q11	559	475	472	472
q12	803	604	602	602
q13	17721	2984	3000	2984
q14	304	295	282	282
q15	524	484	471	471
q16	483	426	427	426
q17	1772	1485	1472	1472
q18	7629	7545	7370	7370
q19	1741	1549	1559	1549
q20	2006	1791	1807	1791
q21	4720	4741	4653	4653
q22	573	512	511	511
Total cold run time: 71783 ms
Total hot run time: 53962 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172508 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2c7c710a69301e079265c70bc18b48d37f3babaa, data reload: false

query1	916	379	370	370
query2	6461	1833	1859	1833
query3	6651	213	216	213
query4	28269	17141	17414	17141
query5	4298	495	484	484
query6	278	184	164	164
query7	4580	296	292	292
query8	242	193	206	193
query9	8480	2374	2376	2374
query10	431	288	283	283
query11	11558	10001	10070	10001
query12	139	88	82	82
query13	1634	370	370	370
query14	10230	7700	7683	7683
query15	225	168	164	164
query16	7998	333	318	318
query17	1792	578	547	547
query18	2042	295	285	285
query19	200	160	159	159
query20	94	84	82	82
query21	207	132	131	131
query22	4467	4265	4098	4098
query23	33696	32894	32965	32894
query24	12078	2923	2947	2923
query25	695	400	396	396
query26	1799	154	153	153
query27	2943	273	276	273
query28	7432	1997	1979	1979
query29	1118	657	634	634
query30	292	151	151	151
query31	975	736	742	736
query32	102	60	57	57
query33	786	312	315	312
query34	887	499	500	499
query35	707	587	606	587
query36	1101	939	944	939
query37	282	86	83	83
query38	2889	2802	2733	2733
query39	873	836	839	836
query40	285	127	128	127
query41	52	48	48	48
query42	119	97	101	97
query43	505	468	468	468
query44	1241	758	752	752
query45	197	166	167	166
query46	1093	744	733	733
query47	1819	1779	1760	1760
query48	387	306	301	301
query49	1192	432	439	432
query50	778	407	412	407
query51	6836	6839	6939	6839
query52	101	100	91	91
query53	368	310	295	295
query54	990	470	470	470
query55	79	77	75	75
query56	315	299	306	299
query57	1176	1058	1051	1051
query58	270	264	271	264
query59	2893	2727	2496	2496
query60	300	280	278	278
query61	96	94	96	94
query62	863	671	647	647
query63	332	299	291	291
query64	10455	2244	1684	1684
query65	3199	3140	3107	3107
query66	1375	342	349	342
query67	15695	15061	14793	14793
query68	8548	560	569	560
query69	709	488	361	361
query70	1224	1053	1189	1053
query71	517	282	283	282
query72	8984	5699	5517	5517
query73	860	326	327	326
query74	5923	5489	5501	5489
query75	5358	2702	2742	2702
query76	5211	1055	952	952
query77	772	306	302	302
query78	9891	8913	10651	8913
query79	11349	518	521	518
query80	886	532	477	477
query81	580	227	219	219
query82	747	139	133	133
query83	337	167	167	167
query84	267	85	92	85
query85	1398	319	298	298
query86	447	340	310	310
query87	3325	3131	3094	3094
query88	4646	2502	2440	2440
query89	540	387	402	387
query90	2008	198	200	198
query91	132	102	105	102
query92	65	51	52	51
query93	6377	496	504	496
query94	1304	217	217	217
query95	405	319	325	319
query96	611	277	270	270
query97	3135	3002	3074	3002
query98	225	201	194	194
query99	1603	1304	1215	1215
Total cold run time: 311242 ms
Total hot run time: 172508 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.59 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2c7c710a69301e079265c70bc18b48d37f3babaa, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.49	0.48	0.50
query6	1.14	0.74	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.56	0.54	0.56
query11	0.15	0.12	0.11
query12	0.16	0.12	0.13
query13	0.58	0.58	0.60
query14	0.75	0.77	0.82
query15	0.85	0.81	0.81
query16	0.36	0.37	0.36
query17	1.06	1.01	1.05
query18	0.22	0.22	0.21
query19	1.81	1.68	1.80
query20	0.01	0.01	0.01
query21	15.41	0.75	0.65
query22	4.23	7.01	2.06
query23	18.27	1.28	1.21
query24	2.14	0.22	0.22
query25	0.16	0.08	0.08
query26	0.30	0.20	0.21
query27	0.45	0.22	0.22
query28	13.28	1.02	1.00
query29	12.61	3.28	3.24
query30	0.25	0.06	0.05
query31	2.88	0.39	0.40
query32	3.26	0.47	0.47
query33	2.91	2.94	2.88
query34	17.05	4.35	4.33
query35	4.42	4.42	4.41
query36	0.66	0.50	0.47
query37	0.18	0.16	0.16
query38	0.16	0.15	0.14
query39	0.04	0.04	0.04
query40	0.16	0.13	0.11
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.79 s
Total hot run time: 30.59 s

@qidaye
Copy link
Contributor Author

qidaye commented Jul 15, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40004 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 13757d4594e27a58842632513678edd2c1faee03, data reload: false

------ Round 1 ----------------------------------
q1	17597	4310	4319	4310
q2	2008	187	183	183
q3	10462	1161	1071	1071
q4	10195	861	865	861
q5	7565	2680	2627	2627
q6	221	139	133	133
q7	960	593	598	593
q8	9222	2044	2063	2044
q9	8834	6574	6549	6549
q10	8831	3763	3764	3763
q11	481	231	235	231
q12	419	219	214	214
q13	17760	2995	2985	2985
q14	273	242	235	235
q15	520	483	500	483
q16	482	400	379	379
q17	947	729	732	729
q18	7998	7377	7557	7377
q19	5479	1421	1373	1373
q20	698	320	335	320
q21	4790	3284	3259	3259
q22	344	285	295	285
Total cold run time: 116086 ms
Total hot run time: 40004 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4333	4199	4206	4199
q2	377	273	269	269
q3	2980	2797	2897	2797
q4	2001	1690	1678	1678
q5	5623	5544	5569	5544
q6	218	129	132	129
q7	2198	1854	1859	1854
q8	3280	3379	3372	3372
q9	8751	8745	8897	8745
q10	4074	3881	3692	3692
q11	627	501	500	500
q12	833	629	618	618
q13	15906	3158	3136	3136
q14	315	279	306	279
q15	539	490	495	490
q16	489	423	438	423
q17	1810	1539	1518	1518
q18	8004	7918	7876	7876
q19	1701	1576	1441	1441
q20	2054	1920	1854	1854
q21	8259	4730	4748	4730
q22	594	478	507	478
Total cold run time: 74966 ms
Total hot run time: 55622 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173498 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 13757d4594e27a58842632513678edd2c1faee03, data reload: false

query1	918	363	362	362
query2	6444	1842	1810	1810
query3	6633	205	220	205
query4	27765	17600	17392	17392
query5	3684	480	465	465
query6	252	184	157	157
query7	4579	298	285	285
query8	245	206	201	201
query9	8550	2382	2359	2359
query10	445	267	275	267
query11	10498	10096	10104	10096
query12	117	87	85	85
query13	1629	371	362	362
query14	10255	7929	7622	7622
query15	229	173	171	171
query16	7520	317	302	302
query17	1820	534	532	532
query18	1877	272	277	272
query19	189	141	149	141
query20	88	83	84	83
query21	206	129	129	129
query22	4338	4076	4107	4076
query23	33992	33846	33528	33528
query24	11064	2856	2950	2856
query25	633	434	387	387
query26	1062	150	153	150
query27	2256	272	276	272
query28	6730	2011	2022	2011
query29	889	664	632	632
query30	256	165	167	165
query31	957	768	779	768
query32	99	57	53	53
query33	761	303	289	289
query34	1027	478	516	478
query35	700	598	593	593
query36	1157	1008	989	989
query37	144	80	87	80
query38	2942	2878	2855	2855
query39	918	851	855	851
query40	208	124	122	122
query41	46	44	45	44
query42	115	103	98	98
query43	516	453	455	453
query44	1193	746	733	733
query45	193	162	165	162
query46	1091	751	729	729
query47	1879	1772	1749	1749
query48	387	292	300	292
query49	845	420	412	412
query50	770	391	383	383
query51	6911	6722	6712	6712
query52	104	99	93	93
query53	357	294	285	285
query54	888	447	431	431
query55	73	72	72	72
query56	307	263	277	263
query57	1145	1067	1084	1067
query58	238	259	259	259
query59	2875	2544	2541	2541
query60	295	270	274	270
query61	96	97	96	96
query62	798	653	623	623
query63	324	291	289	289
query64	9178	2234	7583	2234
query65	3164	3092	3079	3079
query66	715	346	341	341
query67	15436	14974	15283	14974
query68	4494	529	529	529
query69	605	465	348	348
query70	1203	1096	1176	1096
query71	410	276	286	276
query72	7532	5277	6002	5277
query73	755	321	321	321
query74	6264	5697	5641	5641
query75	3409	2692	2660	2660
query76	2992	891	917	891
query77	637	305	302	302
query78	11654	9770	8853	8853
query79	10062	522	505	505
query80	1353	469	473	469
query81	591	232	213	213
query82	587	136	130	130
query83	314	166	161	161
query84	275	87	85	85
query85	770	368	299	299
query86	476	319	312	312
query87	3292	3152	3120	3120
query88	5021	2436	2451	2436
query89	498	385	378	378
query90	2015	189	189	189
query91	126	96	99	96
query92	60	50	51	50
query93	4815	494	487	487
query94	1197	212	212	212
query95	393	314	309	309
query96	628	281	270	270
query97	3177	3075	2992	2992
query98	222	191	190	190
query99	1614	1275	1264	1264
Total cold run time: 291456 ms
Total hot run time: 173498 ms

@qidaye
Copy link
Contributor Author

qidaye commented Jul 15, 2024

run p0

3 similar comments
@qidaye
Copy link
Contributor Author

qidaye commented Jul 15, 2024

run p0

@qidaye
Copy link
Contributor Author

qidaye commented Jul 15, 2024

run p0

@qidaye
Copy link
Contributor Author

qidaye commented Jul 15, 2024

run p0

Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 15, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12 airborne12 merged commit 4339a2f into apache:master Jul 15, 2024
25 of 29 checks passed
@qidaye qidaye deleted the fix_build_index branch July 16, 2024 02:33
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Jul 16, 2024
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Jul 16, 2024
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Jul 16, 2024
xiaokang pushed a commit that referenced this pull request Jul 16, 2024
xiaokang pushed a commit that referenced this pull request Jul 16, 2024
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
…s dropped. (#37646)

## Proposed changes

Issue Number: close #37645
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
1 task
@xiaokang xiaokang removed the doing label Jul 29, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] when dropping indexes, there is lots of NOT_FOUND files logs
5 participants