[improvement](replica) set replica drop write editlog by yujun777 · Pull Request #30345 · apache/doris

yujun777 · 2024-01-25T02:24:20Z

Set replica drop write an editlog. Othewise if user connect to follow fe and execute this cmd, then he show replica will still found replica's IsUserDrop = false, he may confused.

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

yujun777 · 2024-01-25T02:24:28Z

run buildall

yujun777 · 2024-01-25T02:46:53Z

run buildall

doris-robot · 2024-01-25T03:19:55Z

TPC-H: Total hot run time: 38503 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0de39b4811876c540511b9256d51a531420723a7, data reload: false

------ Round 1 ----------------------------------
q1	17717	5155	5209	5155
q2	2039	140	131	131
q3	10643	1147	1182	1147
q4	10212	795	778	778
q5	7724	3096	3140	3096
q6	198	119	125	119
q7	857	490	476	476
q8	9191	1900	1926	1900
q9	7285	6376	6305	6305
q10	8164	3057	3024	3024
q11	412	210	213	210
q12	354	194	189	189
q13	17988	3345	3325	3325
q14	248	214	208	208
q15	549	531	507	507
q16	459	377	372	372
q17	939	486	497	486
q18	7524	6769	6853	6769
q19	1548	1353	1443	1353
q20	594	311	310	310
q21	2749	2404	2348	2348
q22	355	295	311	295
Total cold run time: 107749 ms
Total hot run time: 38503 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5456	5240	5164	5164
q2	324	216	207	207
q3	3353	3218	3207	3207
q4	2062	2071	2053	2053
q5	5993	5771	5906	5771
q6	198	115	114	114
q7	2321	1893	1858	1858
q8	3218	3370	3355	3355
q9	8989	8834	8803	8803
q10	3845	3844	3838	3838
q11	549	442	471	442
q12	813	644	631	631
q13	16935	3215	3123	3123
q14	283	255	263	255
q15	549	513	506	506
q16	503	466	460	460
q17	1869	1861	1871	1861
q18	9630	15939	9821	9821
q19	26265	1572	1526	1526
q20	4643	1934	1931	1931
q21	14312	5528	5293	5293
q22	996	533	554	533
Total cold run time: 113106 ms
Total hot run time: 60752 ms

doris-robot · 2024-01-25T03:31:54Z

TPC-DS: Total hot run time: 187122 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0de39b4811876c540511b9256d51a531420723a7, data reload: false

query1	929	337	323	323
query2	6571	1888	1953	1888
query3	6702	202	198	198
query4	30218	22212	22219	22212
query5	4454	369	375	369
query6	233	156	153	153
query7	4612	261	259	259
query8	215	173	178	173
query9	8532	2616	2628	2616
query10	435	227	241	227
query11	16905	15526	15520	15520
query12	124	71	67	67
query13	1671	373	390	373
query14	10625	6971	7015	6971
query15	216	188	189	188
query16	5792	253	247	247
query17	951	482	479	479
query18	1794	257	254	254
query19	178	132	128	128
query20	72	72	66	66
query21	197	132	130	130
query22	4916	4846	4774	4774
query23	31666	30855	30980	30855
query24	12203	2835	2820	2820
query25	574	310	307	307
query26	1668	146	144	144
query27	3204	284	284	284
query28	7324	1840	1826	1826
query29	1701	629	633	629
query30	278	138	140	138
query31	927	741	752	741
query32	82	50	47	47
query33	705	220	213	213
query34	1129	461	476	461
query35	855	783	794	783
query36	1378	1236	1226	1226
query37	101	61	64	61
query38	3410	3302	3242	3242
query39	1320	1276	1263	1263
query40	348	88	85	85
query41	37	35	35	35
query42	85	80	83	80
query43	521	483	494	483
query44	1086	696	704	696
query45	194	183	170	170
query46	1063	658	652	652
query47	1645	1572	1601	1572
query48	383	309	311	309
query49	1184	295	285	285
query50	678	307	308	307
query51	5338	5141	5190	5141
query52	94	94	82	82
query53	338	267	263	263
query54	241	178	184	178
query55	81	75	81	75
query56	178	169	163	163
query57	1007	915	942	915
query58	178	157	159	157
query59	2774	2708	2761	2708
query60	220	181	186	181
query61	82	89	80	80
query62	616	352	376	352
query63	273	264	259	259
query64	6122	1773	1774	1773
query65	3328	3280	3249	3249
query66	1251	310	308	308
query67	15739	15318	15556	15318
query68	10432	517	526	517
query69	587	315	302	302
query70	1644	1565	1535	1535
query71	10464	10203	10196	10196
query72	4617	2823	2803	2803
query73	1899	320	314	314
query74	6907	6390	6463	6390
query75	4551	2323	2304	2304
query76	6656	1002	1049	1002
query77	650	232	234	232
query78	9032	8930	8496	8496
query79	1312	520	501	501
query80	606	323	315	315
query81	451	201	206	201
query82	205	77	74	74
query83	136	119	116	116
query84	276	65	65	65
query85	1092	315	318	315
query86	377	405	378	378
query87	3478	3387	3335	3335
query88	3662	2193	2198	2193
query89	450	362	365	362
query90	2040	188	186	186
query91	168	126	121	121
query92	49	43	44	43
query93	3521	451	443	443
query94	1232	157	155	155
query95	504	453	444	444
query96	619	317	313	313
query97	4248	4125	4107	4107
query98	206	188	185	185
query99	982	722	648	648
Total cold run time: 303004 ms
Total hot run time: 187122 ms

doris-robot · 2024-01-25T03:37:20Z

ClickBench: Total hot run time: 30.99 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0de39b4811876c540511b9256d51a531420723a7, data reload: false

query1	0.04	0.02	0.02
query2	0.06	0.03	0.02
query3	0.22	0.05	0.05
query4	1.70	0.06	0.07
query5	0.53	0.52	0.53
query6	1.24	0.63	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.54	0.49	0.48
query10	0.57	0.56	0.56
query11	0.11	0.08	0.09
query12	0.10	0.09	0.08
query13	0.60	0.62	0.62
query14	0.79	0.82	0.79
query15	0.79	0.77	0.77
query16	0.39	0.38	0.40
query17	0.99	0.99	1.02
query18	0.24	0.25	0.23
query19	1.83	1.82	1.71
query20	0.01	0.01	0.02
query21	15.41	0.60	0.57
query22	2.56	2.43	2.47
query23	17.65	0.80	0.74
query24	2.58	0.79	1.30
query25	0.37	0.20	0.07
query26	0.56	0.13	0.13
query27	0.06	0.06	0.04
query28	11.86	0.77	0.75
query29	12.49	3.16	3.08
query30	0.53	0.50	0.56
query31	2.79	0.35	0.34
query32	3.37	0.48	0.48
query33	3.17	3.21	3.23
query34	15.65	4.24	4.25
query35	4.32	4.26	4.27
query36	1.12	1.07	1.07
query37	0.07	0.05	0.05
query38	0.04	0.02	0.03
query39	0.03	0.01	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.71 s
Total hot run time: 30.99 s

doris-robot · 2024-01-25T03:40:25Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 0de39b4811876c540511b9256d51a531420723a7 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.7 seconds inserted 10000000 Rows, about 680K ops/s

yujun777 · 2024-01-25T04:03:02Z

run pipelinex_p0

yujun777 · 2024-01-25T04:43:28Z

run buildall

doris-robot · 2024-01-25T05:37:04Z

TPC-H: Total hot run time: 38408 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0de39b4811876c540511b9256d51a531420723a7, data reload: false

------ Round 1 ----------------------------------
q1	18121	4962	5430	4962
q2	2043	141	129	129
q3	10829	1142	1147	1142
q4	10437	755	827	755
q5	7743	3194	3130	3130
q6	195	124	121	121
q7	864	491	480	480
q8	9236	1915	1950	1915
q9	7276	6364	6363	6363
q10	8218	3089	3051	3051
q11	414	199	206	199
q12	356	191	190	190
q13	17987	3359	3402	3359
q14	245	213	215	213
q15	541	506	498	498
q16	424	359	370	359
q17	943	564	493	493
q18	7385	6854	6734	6734
q19	1736	1315	1295	1295
q20	597	304	309	304
q21	2777	2437	2411	2411
q22	348	321	305	305
Total cold run time: 108715 ms
Total hot run time: 38408 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5482	5348	5381	5348
q2	322	221	214	214
q3	3360	3256	3248	3248
q4	2052	2007	2025	2007
q5	6037	5960	5825	5825
q6	202	114	120	114
q7	2274	1873	1865	1865
q8	3218	3357	3376	3357
q9	9040	8871	8816	8816
q10	3914	3793	3748	3748
q11	579	432	436	432
q12	808	637	623	623
q13	16915	3167	3155	3155
q14	293	255	270	255
q15	546	499	499	499
q16	510	468	466	466
q17	1916	1846	1872	1846
q18	9544	17458	9639	9639
q19	26039	1616	1534	1534
q20	4622	1935	1921	1921
q21	14507	5246	5365	5246
q22	955	543	557	543
Total cold run time: 113135 ms
Total hot run time: 60701 ms

doris-robot · 2024-01-25T05:49:01Z

TPC-DS: Total hot run time: 186881 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0de39b4811876c540511b9256d51a531420723a7, data reload: false

query1	935	329	325	325
query2	6583	1953	1936	1936
query3	6696	207	205	205
query4	30257	22273	22381	22273
query5	4469	386	377	377
query6	239	160	154	154
query7	4645	259	264	259
query8	225	180	181	180
query9	8381	2646	2605	2605
query10	429	232	237	232
query11	16760	15478	15602	15478
query12	118	68	68	68
query13	1674	371	402	371
query14	10590	7119	7000	7000
query15	219	183	181	181
query16	5800	263	256	256
query17	940	484	483	483
query18	1792	269	255	255
query19	173	132	137	132
query20	73	73	66	66
query21	180	137	135	135
query22	4944	4779	4808	4779
query23	31618	30826	30868	30826
query24	11616	2805	2852	2805
query25	568	316	305	305
query26	1647	151	155	151
query27	3207	286	287	286
query28	7119	1836	1814	1814
query29	1660	630	617	617
query30	280	138	141	138
query31	956	771	759	759
query32	84	53	49	49
query33	696	215	214	214
query34	1129	453	474	453
query35	892	804	809	804
query36	1318	1190	1176	1176
query37	92	59	57	57
query38	3312	3247	3274	3247
query39	1330	1273	1262	1262
query40	337	95	83	83
query41	39	38	35	35
query42	94	81	84	81
query43	514	437	466	437
query44	1086	699	701	699
query45	191	183	171	171
query46	1072	653	664	653
query47	1650	1599	1563	1563
query48	393	300	299	299
query49	1154	300	302	300
query50	682	319	319	319
query51	5318	5215	5127	5127
query52	92	81	71	71
query53	338	263	262	262
query54	251	181	181	181
query55	83	76	70	70
query56	184	165	160	160
query57	1043	885	927	885
query58	190	154	155	154
query59	2966	2631	2522	2522
query60	207	181	183	181
query61	86	83	84	83
query62	638	358	355	355
query63	285	264	259	259
query64	5712	1818	1790	1790
query65	3356	3296	3260	3260
query66	1276	323	313	313
query67	15775	15201	15370	15201
query68	10404	548	521	521
query69	616	307	296	296
query70	1725	1526	1482	1482
query71	10461	10211	10207	10207
query72	4405	2832	2856	2832
query73	1913	323	323	323
query74	7534	6407	6318	6318
query75	4538	2376	2346	2346
query76	6539	1060	985	985
query77	643	227	229	227
query78	9022	8753	8594	8594
query79	1073	501	500	500
query80	753	314	323	314
query81	463	208	208	208
query82	208	87	78	78
query83	173	113	121	113
query84	280	68	68	68
query85	1113	336	328	328
query86	381	401	358	358
query87	3451	3375	3342	3342
query88	3513	2190	2190	2190
query89	436	380	344	344
query90	1993	188	191	188
query91	159	124	137	124
query92	52	46	44	44
query93	3220	455	424	424
query94	1195	160	155	155
query95	508	459	461	459
query96	633	311	326	311
query97	4292	4103	4183	4103
query98	212	195	192	192
query99	1041	680	713	680
Total cold run time: 301607 ms
Total hot run time: 186881 ms

doris-robot · 2024-01-25T05:54:30Z

ClickBench: Total hot run time: 31.16 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0de39b4811876c540511b9256d51a531420723a7, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.02
query3	0.22	0.06	0.06
query4	1.66	0.08	0.08
query5	0.54	0.53	0.53
query6	1.34	0.62	0.64
query7	0.03	0.02	0.01
query8	0.04	0.02	0.02
query9	0.54	0.51	0.51
query10	0.56	0.56	0.54
query11	0.12	0.09	0.09
query12	0.12	0.11	0.09
query13	0.61	0.61	0.61
query14	0.77	0.81	0.82
query15	0.80	0.77	0.76
query16	0.39	0.39	0.39
query17	1.00	1.01	1.03
query18	0.24	0.24	0.24
query19	1.85	1.82	1.80
query20	0.02	0.01	0.01
query21	15.41	0.60	0.56
query22	2.70	2.69	1.83
query23	17.53	0.80	0.77
query24	3.06	1.02	1.09
query25	0.42	0.18	0.23
query26	0.52	0.14	0.13
query27	0.06	0.06	0.05
query28	10.75	0.78	0.77
query29	12.49	3.15	3.32
query30	0.57	0.45	0.51
query31	2.78	0.35	0.35
query32	3.35	0.49	0.49
query33	3.24	3.24	3.22
query34	15.84	4.30	4.29
query35	4.39	4.33	4.34
query36	1.12	1.07	1.09
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.15	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.62 s
Total hot run time: 31.16 s

doris-robot · 2024-01-25T05:57:38Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 0de39b4811876c540511b9256d51a531420723a7 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.8 seconds inserted 10000000 Rows, about 675K ops/s

deardeng

LGTM

github-actions · 2024-01-25T06:46:52Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

github-actions · 2024-01-25T07:01:25Z

PR approved by at least one committer and no changes requested.

* set replica drop write editlog * update test

…pache#34033)

* set replica drop write editlog * update test

set replica drop write editlog

042cbd7

update test

0de39b4

deardeng approved these changes Jan 25, 2024

View reviewed changes

github-actions Bot added the reviewed label Jan 25, 2024

dataroaring approved these changes Jan 25, 2024

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jan 25, 2024

yiguolei merged commit 7e9f410 into apache:master Jan 25, 2024

yiguolei pushed a commit that referenced this pull request Jan 25, 2024

[improvement](replica) set replica drop write editlog (#30345)

4681958

* set replica drop write editlog * update test

xiaokang added the dev/2.0.5 label Jan 26, 2024

yujun777 added a commit to yujun777/doris that referenced this pull request Apr 23, 2024

[improvement](replica) set replica drop write editlog (apache#30345)

2f427bf

* set replica drop write editlog * update test

yujun777 mentioned this pull request Apr 23, 2024

[improvement](replica) set replica drop write editlog #30345 #34033

Merged

dataroaring pushed a commit that referenced this pull request Apr 24, 2024

[improvement](replica) set replica drop write editlog (#30345) (#34033)

d1e614a

dataroaring added dev/2.0.10-merged and removed dev/2.0.5 labels Apr 24, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[improvement](replica) set replica drop write editlog (apache#30345) (a…

7a31171

…pache#34033)

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026

[improvement](replica) set replica drop write editlog (apache#30345)

098a9b5

* set replica drop write editlog * update test

Conversation

yujun777 commented Jan 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

yujun777 commented Jan 25, 2024

Uh oh!

yujun777 commented Jan 25, 2024

Uh oh!

doris-robot commented Jan 25, 2024

Uh oh!

doris-robot commented Jan 25, 2024

Uh oh!

doris-robot commented Jan 25, 2024

Uh oh!

doris-robot commented Jan 25, 2024

Uh oh!

yujun777 commented Jan 25, 2024

Uh oh!

yujun777 commented Jan 25, 2024

Uh oh!

doris-robot commented Jan 25, 2024

Uh oh!

doris-robot commented Jan 25, 2024

Uh oh!

doris-robot commented Jan 25, 2024

Uh oh!

doris-robot commented Jan 25, 2024

Uh oh!

deardeng left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jan 25, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jan 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

yujun777 commented Jan 25, 2024 •

edited

Loading