[fix](fe) Fix drop table write too many times editlog by swjtu-zhanglei · Pull Request #29966 · apache/doris

swjtu-zhanglei · 2024-01-15T03:33:35Z

No description provided.

swjtu-zhanglei · 2024-01-15T03:34:22Z

run buildall

doris-robot · 2024-01-15T04:12:27Z

TPC-H: Total hot run time: 40031 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit afe913c9534fd1f1d721ed1c5a696fa30014e4c0, data reload: false

------ Round 1 ----------------------------------
q1	17927	6320	5840	5840
q2	2050	152	134	134
q3	10848	1146	1250	1146
q4	10472	855	890	855
q5	8163	3246	3203	3203
q6	208	128	129	128
q7	903	524	529	524
q8	9512	2045	2039	2039
q9	7925	6566	6459	6459
q10	8213	3071	3100	3071
q11	410	201	221	201
q12	366	202	201	201
q13	18093	3417	3419	3417
q14	253	222	222	222
q15	568	519	509	509
q16	444	392	390	390
q17	962	506	496	496
q18	7806	7153	6785	6785
q19	1629	1356	1464	1356
q20	557	300	343	300
q21	2843	2507	2437	2437
q22	355	318	344	318
Total cold run time: 110507 ms
Total hot run time: 40031 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5997	5731	5681	5681
q2	367	222	223	222
q3	3341	3176	3147	3147
q4	2200	2163	1986	1986
q5	6128	5967	6027	5967
q6	206	124	122	122
q7	2374	1973	1986	1973
q8	3679	3746	3752	3746
q9	9594	16935	9903	9903
q10	9301	3747	3827	3747
q11	602	457	466	457
q12	1194	647	651	647
q13	17107	3235	3253	3235
q14	308	246	279	246
q15	592	517	514	514
q16	530	528	489	489
q17	2302	1903	2068	1903
q18	9763	9414	11925	9414
q19	14019	1633	1631	1631
q20	4120	1948	1953	1948
q21	14732	6158	5700	5700
q22	2437	590	587	587
Total cold run time: 110893 ms
Total hot run time: 63265 ms

doris-robot · 2024-01-15T04:13:45Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.83 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 21.6 seconds inserted 10000000 Rows, about 462K ops/s
storage size: 17183583537 Bytes

doris-robot · 2024-01-15T04:24:12Z

TPC-DS: Total hot run time: 179042 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit afe913c9534fd1f1d721ed1c5a696fa30014e4c0, data reload: false

query1	925	338	328	328
query2	6716	1911	1928	1911
query3	6716	207	208	207
query4	36067	22330	22276	22276
query5	6935	599	586	586
query6	267	176	198	176
query7	4618	272	267	267
query8	230	200	203	200
query9	9174	2865	2725	2725
query10	617	252	246	246
query11	16339	15282	15223	15223
query12	126	72	71	71
query13	1707	394	388	388
query14	11397	7460	7397	7397
query15	216	202	187	187
query16	4641	250	246	246
query17	1012	489	471	471
query18	1805	271	260	260
query19	185	145	141	141
query20	80	80	77	77
query21	195	139	131	131
query22	5024	4846	4716	4716
query23	31937	31155	30902	30902
query24	12467	2797	2783	2783
query25	580	314	325	314
query26	1902	146	153	146
query27	3258	277	275	275
query28	7209	1874	1878	1874
query29	2232	693	656	656
query30	282	136	142	136
query31	956	756	771	756
query32	76	62	62	62
query33	719	243	249	243
query34	1174	464	464	464
query35	918	758	764	758
query36	1418	1291	1263	1263
query37	185	68	68	68
query38	3395	3238	3315	3238
query39	1338	1267	1258	1258
query40	340	101	98	98
query41	41	35	36	35
query42	99	96	97	96
query43	546	542	522	522
query44	1060	706	722	706
query45	209	188	178	178
query46	1060	638	668	638
query47	1729	1630	1627	1627
query48	390	308	312	308
query49	1251	313	326	313
query50	704	324	326	324
query51	5435	5168	5206	5168
query52	102	88	97	88
query53	352	284	299	284
query54	886	457	462	457
query55	94	92	91	91
query56	210	194	200	194
query57	1062	947	1003	947
query58	207	194	200	194
query59	2940	2787	2795	2787
query60	228	227	235	227
query61	87	82	89	82
query62	681	430	417	417
query63	314	291	273	273
query64	6248	1721	1656	1656
query65	3332	3294	3259	3259
query66	1456	337	339	337
query67	15834	15280	15163	15163
query68	11988	527	502	502
query69	633	338	337	337
query70	1701	1494	1505	1494
query71	562	233	257	233
query72	5033	2841	2849	2841
query73	2481	321	312	312
query74	7104	6416	6444	6416
query75	5000	2287	2321	2287
query76	6261	936	1049	936
query77	669	288	274	274
query78	9148	8771	8686	8686
query79	1015	501	485	485
query80	533	347	347	347
query81	441	206	205	205
query82	210	88	92	88
query83	140	120	127	120
query84	279	74	71	71
query85	1045	334	333	333
query86	378	399	370	370
query87	3535	3386	3395	3386
query88	3058	2211	2213	2211
query89	470	397	378	378
query90	2029	210	199	199
query91	158	134	134	134
query92	63	58	56	56
query93	1541	410	401	401
query94	1258	181	187	181
query95	526	514	479	479
query96	618	314	312	312
query97	4287	4180	4248	4180
query98	197	188	183	183
query99	1076	743	710	710
Total cold run time: 305372 ms
Total hot run time: 179042 ms

doris-robot · 2024-01-15T04:29:40Z

ClickBench: Total hot run time: 31.08 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit afe913c9534fd1f1d721ed1c5a696fa30014e4c0, data reload: false

query1	0.05	0.06	0.06
query2	0.06	0.02	0.02
query3	0.24	0.12	0.11
query4	1.78	0.12	0.12
query5	0.54	0.54	0.54
query6	1.32	0.64	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.49
query10	0.56	0.56	0.55
query11	0.12	0.09	0.09
query12	0.12	0.09	0.10
query13	0.62	0.61	0.61
query14	0.77	0.79	0.80
query15	0.83	0.79	0.82
query16	0.40	0.41	0.38
query17	1.04	1.02	1.04
query18	0.25	0.25	0.25
query19	1.92	1.77	1.78
query20	0.02	0.01	0.01
query21	15.44	0.56	0.56
query22	2.96	2.09	2.04
query23	17.24	0.92	0.82
query24	2.55	0.71	0.39
query25	1.91	0.16	0.15
query26	0.14	0.13	0.13
query27	0.16	0.15	0.15
query28	11.76	0.81	0.79
query29	12.55	3.22	3.22
query30	0.56	0.49	0.49
query31	2.78	0.34	0.36
query32	3.38	0.50	0.50
query33	3.31	3.32	3.28
query34	15.81	4.35	4.26
query35	4.30	4.26	4.24
query36	1.12	1.09	1.10
query37	0.07	0.05	0.05
query38	0.04	0.02	0.02
query39	0.03	0.01	0.01
query40	0.17	0.13	0.13
query41	0.08	0.01	0.01
query42	0.03	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 107.68 s
Total hot run time: 31.08 s

doris-robot · 2024-01-15T04:32:45Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit afe913c9534fd1f1d721ed1c5a696fa30014e4c0 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.8 seconds inserted 10000000 Rows, about 675K ops/s

w41ter · 2024-01-16T02:20:18Z

fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java

                }
                case OperationType.OP_DELETE_TABLE_STATS: {
-                    env.getAnalysisManager().replayTableStatsDeletion((TableStatsDeletionLog) journal.getData());
+                    LOG.info("replay delete table stat: {}", ((TableStatsDeletionLog) journal.getData()).id);


Without replay table stats deletion, how to clear a follower's table stats?

Without replay table stats deletion, how to clear a follower's table stats?

Thanks for you comment, i hava updated the code.

swjtu-zhanglei · 2024-01-16T02:29:58Z

run buildall

swjtu-zhanglei · 2024-01-16T02:32:01Z

run buildall

doris-robot · 2024-01-16T03:00:31Z

TPC-H: Total hot run time: 40016 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ddfa1b51d02430c21330260ee3bc8b715ad1aae0, data reload: false

------ Round 1 ----------------------------------
q1	18862	6508	6058	6058
q2	2469	150	170	150
q3	11340	1195	1273	1195
q4	10803	808	874	808
q5	8396	3320	3153	3153
q6	212	126	125	125
q7	889	499	484	484
q8	9237	1972	1986	1972
q9	7342	6424	6408	6408
q10	8243	3088	3098	3088
q11	426	204	218	204
q12	362	204	203	203
q13	18106	3405	3409	3405
q14	249	220	226	220
q15	547	525	516	516
q16	439	397	379	379
q17	945	506	489	489
q18	7683	7009	6882	6882
q19	1637	1256	1401	1256
q20	535	309	316	309
q21	2833	2401	2444	2401
q22	379	319	311	311
Total cold run time: 111934 ms
Total hot run time: 40016 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5994	5732	5608	5608
q2	348	223	226	223
q3	3311	3094	3097	3094
q4	2211	2094	2046	2046
q5	6172	5956	5952	5952
q6	209	122	122	122
q7	2476	1965	1960	1960
q8	3561	3743	3738	3738
q9	9711	25395	9584	9584
q10	4327	3839	3784	3784
q11	833	464	445	445
q12	1240	661	631	631
q13	17096	3255	3230	3230
q14	372	271	269	269
q15	569	519	521	519
q16	526	487	475	475
q17	9233	2036	2014	2014
q18	9550	9625	11181	9625
q19	12944	1648	1630	1630
q20	3428	1969	1932	1932
q21	13596	5633	5899	5633
q22	1040	582	582	582
Total cold run time: 108747 ms
Total hot run time: 63096 ms

doris-robot · 2024-01-16T03:12:11Z

TPC-DS: Total hot run time: 177770 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ddfa1b51d02430c21330260ee3bc8b715ad1aae0, data reload: false

query1	927	329	333	329
query2	6735	1962	1848	1848
query3	6753	206	210	206
query4	34973	21976	21980	21976
query5	6942	584	532	532
query6	249	180	177	177
query7	4620	277	275	275
query8	240	191	205	191
query9	9198	2727	2677	2677
query10	612	237	239	237
query11	17121	15422	15191	15191
query12	122	71	66	66
query13	1698	373	400	373
query14	11302	7337	7309	7309
query15	218	185	178	178
query16	4624	256	250	250
query17	993	494	486	486
query18	1801	268	275	268
query19	183	150	145	145
query20	82	75	68	68
query21	198	130	137	130
query22	5007	4736	4648	4648
query23	31940	30982	31005	30982
query24	12513	2802	2834	2802
query25	584	336	321	321
query26	1882	147	153	147
query27	3233	279	272	272
query28	7096	1866	1846	1846
query29	2260	691	691	691
query30	287	141	138	138
query31	946	762	766	762
query32	79	62	61	61
query33	717	241	227	227
query34	1189	465	465	465
query35	868	755	736	736
query36	1416	1293	1302	1293
query37	104	69	65	65
query38	3428	3309	3279	3279
query39	1306	1261	1269	1261
query40	344	99	88	88
query41	39	35	35	35
query42	93	98	96	96
query43	567	514	514	514
query44	1086	699	708	699
query45	205	187	180	180
query46	1069	668	619	619
query47	1617	1608	1594	1594
query48	416	317	329	317
query49	1237	312	318	312
query50	693	320	319	319
query51	5369	5201	5230	5201
query52	98	100	85	85
query53	358	278	285	278
query54	890	446	469	446
query55	98	86	82	82
query56	209	191	203	191
query57	1014	974	935	935
query58	227	185	213	185
query59	2941	2769	2594	2594
query60	229	208	221	208
query61	88	82	86	82
query62	703	399	408	399
query63	314	277	293	277
query64	6243	1686	1680	1680
query65	3356	3263	3258	3258
query66	1445	346	319	319
query67	15715	14970	15365	14970
query68	10923	507	504	504
query69	654	328	328	328
query70	1654	1539	1561	1539
query71	560	237	223	223
query72	5065	2832	2908	2832
query73	2102	306	311	306
query74	6992	6486	6396	6396
query75	4754	2351	2287	2287
query76	6275	861	1050	861
query77	739	258	293	258
query78	9148	8818	8614	8614
query79	1005	505	494	494
query80	552	338	352	338
query81	446	200	204	200
query82	204	89	88	88
query83	144	121	119	119
query84	278	71	72	71
query85	1047	342	325	325
query86	372	376	368	368
query87	3557	3367	3370	3367
query88	3185	2188	2175	2175
query89	456	393	398	393
query90	2008	202	198	198
query91	155	128	130	128
query92	59	56	52	52
query93	993	419	441	419
query94	1137	182	177	177
query95	521	470	469	469
query96	620	327	313	313
query97	4355	4138	4182	4138
query98	214	194	184	184
query99	1013	735	742	735
Total cold run time: 302296 ms
Total hot run time: 177770 ms

doris-robot · 2024-01-16T03:17:37Z

ClickBench: Total hot run time: 30.66 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ddfa1b51d02430c21330260ee3bc8b715ad1aae0, data reload: false

query1	0.06	0.06	0.05
query2	0.06	0.03	0.03
query3	0.25	0.12	0.10
query4	1.77	0.12	0.12
query5	0.55	0.53	0.53
query6	1.32	0.64	0.64
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.54	0.51	0.50
query10	0.54	0.55	0.55
query11	0.12	0.09	0.08
query12	0.11	0.10	0.09
query13	0.60	0.61	0.60
query14	0.77	0.77	0.79
query15	0.79	0.76	0.76
query16	0.37	0.41	0.38
query17	1.02	1.03	1.00
query18	0.24	0.24	0.28
query19	1.85	1.80	1.75
query20	0.01	0.01	0.01
query21	15.42	0.57	0.56
query22	2.27	2.51	2.24
query23	17.25	0.83	0.77
query24	2.58	1.66	0.43
query25	0.40	0.20	0.06
query26	0.64	0.15	0.14
query27	0.06	0.05	0.04
query28	11.18	0.79	0.82
query29	12.56	3.14	3.14
query30	0.52	0.49	0.50
query31	2.78	0.35	0.35
query32	3.41	0.48	0.49
query33	3.20	3.21	3.26
query34	15.91	4.25	4.20
query35	4.32	4.24	4.26
query36	1.11	1.05	1.07
query37	0.07	0.05	0.05
query38	0.04	0.02	0.02
query39	0.03	0.02	0.01
query40	0.16	0.12	0.12
query41	0.08	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.06 s
Total hot run time: 30.66 s

doris-robot · 2024-01-16T03:20:41Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ddfa1b51d02430c21330260ee3bc8b715ad1aae0 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

dataroaring

LGTM

github-actions · 2024-01-17T08:23:46Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-01-17T08:23:49Z

PR approved by anyone and no changes requested.

github-actions bot added the meta-change label Jan 15, 2024

w41ter reviewed Jan 16, 2024

View reviewed changes

swjtu-zhanglei force-pushed the master-fix4 branch from afe913c to 3a97f17 Compare January 16, 2024 02:29

[fix](fe) Fix drop table write too many times editlog

ddfa1b5

swjtu-zhanglei force-pushed the master-fix4 branch from 3a97f17 to ddfa1b5 Compare January 16, 2024 02:30

dataroaring approved these changes Jan 17, 2024

View reviewed changes

w41ter approved these changes Jan 17, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 17, 2024

github-actions bot added the reviewed label Jan 17, 2024

dataroaring added the dev/2.1.0 label Jan 17, 2024

dataroaring merged commit c7784b1 into apache:master Jan 17, 2024

yiguolei pushed a commit that referenced this pull request Jan 18, 2024

[fix](fe) Fix drop table write too many times editlog (#29966)

e73c0f4

hello-stephen pushed a commit to hello-stephen/doris that referenced this pull request Jan 19, 2024

[fix](fe) Fix drop table write too many times editlog (apache#29966)

fb8e052

seawinde pushed a commit to seawinde/doris that referenced this pull request Jan 22, 2024

[fix](fe) Fix drop table write too many times editlog (apache#29966)

67a0104

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Conversation

swjtu-zhanglei commented Jan 15, 2024

Uh oh!

swjtu-zhanglei commented Jan 15, 2024

Uh oh!

doris-robot commented Jan 15, 2024

Uh oh!

doris-robot commented Jan 15, 2024

Uh oh!

doris-robot commented Jan 15, 2024

Uh oh!

doris-robot commented Jan 15, 2024

Uh oh!

doris-robot commented Jan 15, 2024

Uh oh!

w41ter Jan 16, 2024

Choose a reason for hiding this comment

Uh oh!

swjtu-zhanglei Jan 16, 2024

Choose a reason for hiding this comment

Uh oh!

swjtu-zhanglei commented Jan 16, 2024

Uh oh!

swjtu-zhanglei commented Jan 16, 2024

Uh oh!

doris-robot commented Jan 16, 2024

Uh oh!

doris-robot commented Jan 16, 2024

Uh oh!

doris-robot commented Jan 16, 2024

Uh oh!

doris-robot commented Jan 16, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 17, 2024

Uh oh!

github-actions bot commented Jan 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants