[chore](binlog) allow reading binlogs from follower #50714

w41ter · 2025-05-08T12:04:28Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: https://github.com/selectdb/ccr-syncer/pull/614/files

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-05-08T12:04:33Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

w41ter · 2025-05-08T12:05:55Z

run buildall

doris-robot · 2025-05-08T12:34:41Z

TeamCity cloud ut coverage result:
Function Coverage: 83.18% (1103/1326)
Line Coverage: 66.09% (18561/28084)
Region Coverage: 65.73% (9250/14073)
Branch Coverage: 55.51% (4974/8960)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6a7f84d0a926563b7d8409388135cea18e7a9c32_6a7f84d0a926563b7d8409388135cea18e7a9c32_cloud/report/index.html

fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java

doris-robot · 2025-05-08T13:56:59Z

TPC-H: Total hot run time: 33651 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6a7f84d0a926563b7d8409388135cea18e7a9c32, data reload: false

------ Round 1 ----------------------------------
q1	26418	5021	4969	4969
q2	2135	279	183	183
q3	10381	1247	688	688
q4	10247	1003	516	516
q5	7657	2354	2381	2354
q6	181	162	131	131
q7	916	738	595	595
q8	9317	1286	1067	1067
q9	6774	5121	5033	5033
q10	6862	2287	1879	1879
q11	499	286	274	274
q12	357	347	216	216
q13	17777	3628	3071	3071
q14	233	224	214	214
q15	526	469	485	469
q16	432	426	376	376
q17	602	869	358	358
q18	7677	7215	7145	7145
q19	1697	951	541	541
q20	325	330	214	214
q21	3865	3398	2395	2395
q22	1053	1048	963	963
Total cold run time: 115931 ms
Total hot run time: 33651 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5510	5051	5050	5050
q2	238	337	233	233
q3	2207	2618	2311	2311
q4	1366	1754	1458	1458
q5	4438	4438	4415	4415
q6	218	169	125	125
q7	1946	1955	1799	1799
q8	2611	2652	2646	2646
q9	7266	7158	6948	6948
q10	3014	3151	2710	2710
q11	574	516	472	472
q12	664	771	613	613
q13	3502	3854	3268	3268
q14	296	291	288	288
q15	543	479	476	476
q16	439	472	422	422
q17	1139	1556	1402	1402
q18	7835	7606	7432	7432
q19	788	793	831	793
q20	1989	2081	1808	1808
q21	4955	4812	4725	4725
q22	1083	1055	1050	1050
Total cold run time: 52621 ms
Total hot run time: 50444 ms

doris-robot · 2025-05-08T14:09:09Z

TPC-DS: Total hot run time: 189417 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6a7f84d0a926563b7d8409388135cea18e7a9c32, data reload: false

query1	1398	1095	1065	1065
query2	6369	1806	1779	1779
query3	10996	4434	4517	4434
query4	52031	26200	23139	23139
query5	5066	595	454	454
query6	358	198	195	195
query7	5117	495	287	287
query8	315	279	238	238
query9	5667	2571	2559	2559
query10	451	313	263	263
query11	15072	15155	14927	14927
query12	157	111	106	106
query13	1080	517	418	418
query14	10076	6257	6151	6151
query15	206	203	176	176
query16	7116	668	538	538
query17	1126	740	601	601
query18	1610	420	321	321
query19	203	194	169	169
query20	132	120	119	119
query21	208	128	105	105
query22	4584	4349	4288	4288
query23	34167	33582	33632	33582
query24	6653	2431	2404	2404
query25	471	466	397	397
query26	721	271	150	150
query27	2374	495	335	335
query28	2952	2126	2110	2110
query29	573	544	422	422
query30	265	216	191	191
query31	877	827	766	766
query32	72	59	62	59
query33	465	363	312	312
query34	757	842	533	533
query35	795	836	770	770
query36	928	995	905	905
query37	113	97	76	76
query38	4269	4403	4241	4241
query39	1499	1418	1422	1418
query40	222	121	114	114
query41	55	56	54	54
query42	119	114	108	108
query43	491	510	480	480
query44	1308	811	810	810
query45	185	184	173	173
query46	845	1040	625	625
query47	1852	1888	1836	1836
query48	382	404	306	306
query49	692	516	449	449
query50	654	705	412	412
query51	4281	4277	4141	4141
query52	110	108	97	97
query53	228	259	183	183
query54	585	587	510	510
query55	83	81	82	81
query56	308	288	299	288
query57	1155	1171	1131	1131
query58	259	255	242	242
query59	2784	2741	2736	2736
query60	326	314	294	294
query61	135	131	140	131
query62	760	759	691	691
query63	237	194	189	189
query64	1993	1059	736	736
query65	4414	4224	4225	4224
query66	717	388	301	301
query67	15719	15334	15517	15334
query68	6875	882	502	502
query69	526	337	258	258
query70	1182	1143	1092	1092
query71	500	310	282	282
query72	5735	2383	5073	2383
query73	1471	759	341	341
query74	9187	8894	8663	8663
query75	4088	3179	2699	2699
query76	4253	1200	744	744
query77	752	367	272	272
query78	9921	9983	9198	9198
query79	3750	799	543	543
query80	613	554	437	437
query81	464	253	215	215
query82	493	131	99	99
query83	289	246	227	227
query84	290	101	86	86
query85	765	349	339	339
query86	373	315	271	271
query87	4285	4426	4274	4274
query88	3406	2162	2220	2162
query89	430	310	272	272
query90	1930	200	207	200
query91	152	140	111	111
query92	76	58	56	56
query93	2887	941	572	572
query94	656	409	294	294
query95	358	295	285	285
query96	476	562	271	271
query97	3129	3192	3071	3071
query98	221	208	193	193
query99	1440	1410	1326	1326
Total cold run time: 299161 ms
Total hot run time: 189417 ms

doris-robot · 2025-05-08T14:14:34Z

ClickBench: Total hot run time: 29.53 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6a7f84d0a926563b7d8409388135cea18e7a9c32, data reload: false

query1	0.04	0.03	0.04
query2	0.15	0.10	0.10
query3	0.35	0.20	0.20
query4	1.59	0.20	0.08
query5	0.56	0.55	0.55
query6	1.19	0.72	0.72
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.61	0.52	0.52
query10	0.56	0.58	0.58
query11	0.25	0.12	0.12
query12	0.26	0.13	0.13
query13	0.63	0.61	0.60
query14	0.79	0.82	0.82
query15	0.95	0.88	0.87
query16	0.38	0.38	0.36
query17	1.04	1.03	1.03
query18	0.17	0.19	0.18
query19	1.95	1.82	1.80
query20	0.02	0.01	0.01
query21	15.40	0.97	0.66
query22	0.92	1.07	0.80
query23	14.70	1.51	0.75
query24	5.45	0.59	0.28
query25	0.16	0.10	0.08
query26	0.56	0.23	0.18
query27	0.08	0.09	0.08
query28	10.96	1.18	0.56
query29	12.59	4.04	3.34
query30	0.28	0.08	0.06
query31	2.82	0.61	0.42
query32	3.22	0.59	0.50
query33	3.12	3.02	3.12
query34	16.37	5.10	4.48
query35	4.43	4.48	4.44
query36	0.65	0.50	0.48
query37	0.19	0.17	0.16
query38	0.17	0.16	0.15
query39	0.06	0.05	0.04
query40	0.19	0.17	0.16
query41	0.11	0.06	0.05
query42	0.06	0.05	0.05
query43	0.06	0.06	0.04
Total cold run time: 104.12 s
Total hot run time: 29.53 s

hello-stephen · 2025-05-08T15:32:57Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.68% (14895/26751)
Line Coverage	44.49% (131348/295259)
Region Coverage	43.39% (66180/152530)
Branch Coverage	38.07% (33798/88790)

hello-stephen · 2025-05-08T17:40:09Z

BE Regression && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.29% (20812/26248)
Line Coverage	72.52% (213744/294757)
Region Coverage	70.66% (126410/178890)
Branch Coverage	64.31% (65160/101322)

w41ter · 2025-05-09T02:07:13Z

run buildall

doris-robot · 2025-05-09T02:35:45Z

TeamCity cloud ut coverage result:
Function Coverage: 83.18% (1103/1326)
Line Coverage: 66.10% (18563/28084)
Region Coverage: 65.67% (9242/14073)
Branch Coverage: 55.47% (4970/8960)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5a4ded288f61ccd15f23875207f09ab2633b2039_5a4ded288f61ccd15f23875207f09ab2633b2039_cloud/report/index.html

doris-robot · 2025-05-09T02:48:04Z

TPC-H: Total hot run time: 34036 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5a4ded288f61ccd15f23875207f09ab2633b2039, data reload: false

------ Round 1 ----------------------------------
q1	26551	5061	5055	5055
q2	2087	308	197	197
q3	10601	1273	722	722
q4	10254	1018	516	516
q5	7897	2400	2328	2328
q6	185	166	133	133
q7	932	721	609	609
q8	9304	1279	1111	1111
q9	6920	5104	5060	5060
q10	6842	2342	1896	1896
q11	474	288	272	272
q12	355	353	213	213
q13	17767	3681	3127	3127
q14	227	226	219	219
q15	565	485	480	480
q16	428	423	375	375
q17	608	854	368	368
q18	7873	7266	7180	7180
q19	1434	955	563	563
q20	346	351	223	223
q21	4042	2661	2426	2426
q22	1058	982	963	963
Total cold run time: 116750 ms
Total hot run time: 34036 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5101	5100	5083	5083
q2	242	322	230	230
q3	2211	2678	2368	2368
q4	1391	1831	1436	1436
q5	4583	4440	4388	4388
q6	222	171	127	127
q7	2032	1942	1820	1820
q8	2620	2541	2496	2496
q9	7196	7076	7063	7063
q10	3005	3191	2723	2723
q11	592	508	491	491
q12	656	783	597	597
q13	3558	3990	3244	3244
q14	272	295	289	289
q15	513	466	474	466
q16	425	491	460	460
q17	1178	1482	1418	1418
q18	7627	7555	7329	7329
q19	808	823	833	823
q20	1986	2033	1889	1889
q21	5155	4612	4618	4612
q22	1087	990	974	974
Total cold run time: 52460 ms
Total hot run time: 50326 ms

doris-robot · 2025-05-09T02:59:38Z

TPC-DS: Total hot run time: 186099 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5a4ded288f61ccd15f23875207f09ab2633b2039, data reload: false

query1	1032	469	518	469
query2	6560	1787	1794	1787
query3	6756	224	221	221
query4	26214	23361	23139	23139
query5	4301	620	445	445
query6	298	209	182	182
query7	4621	488	277	277
query8	293	249	233	233
query9	8623	2542	2563	2542
query10	483	366	286	286
query11	15382	15016	14753	14753
query12	159	108	110	108
query13	1660	513	389	389
query14	8727	6264	6122	6122
query15	209	200	177	177
query16	7132	652	465	465
query17	913	728	558	558
query18	1961	405	304	304
query19	193	184	157	157
query20	120	114	114	114
query21	257	127	103	103
query22	4043	4197	4102	4102
query23	33883	32900	32903	32900
query24	8530	2421	2384	2384
query25	545	455	387	387
query26	1240	264	152	152
query27	2760	487	328	328
query28	4380	2091	2079	2079
query29	759	551	431	431
query30	281	218	186	186
query31	914	865	767	767
query32	73	65	63	63
query33	552	363	303	303
query34	823	847	529	529
query35	793	807	774	774
query36	973	998	910	910
query37	120	102	89	89
query38	4247	4200	4205	4200
query39	1623	1400	1410	1400
query40	219	118	105	105
query41	58	55	52	52
query42	119	111	111	111
query43	510	500	458	458
query44	1265	780	777	777
query45	182	171	170	170
query46	825	1018	631	631
query47	1710	1798	1732	1732
query48	376	401	295	295
query49	775	530	426	426
query50	643	676	394	394
query51	4101	4212	4127	4127
query52	108	104	98	98
query53	238	255	181	181
query54	586	564	498	498
query55	88	79	78	78
query56	309	301	281	281
query57	1123	1133	1101	1101
query58	263	264	257	257
query59	2646	2659	2589	2589
query60	331	311	297	297
query61	131	126	123	123
query62	803	727	680	680
query63	234	189	190	189
query64	4423	1088	766	766
query65	4322	4237	4251	4237
query66	1195	416	317	317
query67	15860	15644	15641	15641
query68	8354	872	506	506
query69	467	300	268	268
query70	1226	1081	1057	1057
query71	471	319	293	293
query72	5401	4784	4872	4784
query73	734	631	343	343
query74	8865	9005	8883	8883
query75	3832	3202	2715	2715
query76	3620	1191	754	754
query77	789	378	285	285
query78	9965	10088	9236	9236
query79	3241	813	570	570
query80	662	520	455	455
query81	478	257	223	223
query82	485	130	99	99
query83	276	260	231	231
query84	288	107	88	88
query85	852	357	315	315
query86	350	330	288	288
query87	4552	4485	4354	4354
query88	2937	2240	2230	2230
query89	434	309	283	283
query90	1947	225	210	210
query91	204	150	114	114
query92	83	61	59	59
query93	2025	956	583	583
query94	678	415	301	301
query95	380	292	284	284
query96	488	567	277	277
query97	3197	3200	3094	3094
query98	236	208	204	204
query99	1442	1394	1287	1287
Total cold run time: 274835 ms
Total hot run time: 186099 ms

doris-robot · 2025-05-09T03:04:58Z

ClickBench: Total hot run time: 29.42 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5a4ded288f61ccd15f23875207f09ab2633b2039, data reload: false

query1	0.04	0.03	0.04
query2	0.12	0.11	0.11
query3	0.26	0.19	0.20
query4	1.60	0.19	0.19
query5	0.61	0.58	0.61
query6	1.17	0.72	0.70
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.58	0.53	0.50
query10	0.58	0.57	0.56
query11	0.16	0.10	0.11
query12	0.15	0.12	0.11
query13	0.61	0.61	0.60
query14	0.77	0.80	0.81
query15	0.88	0.84	0.85
query16	0.38	0.38	0.40
query17	1.06	1.01	1.01
query18	0.20	0.20	0.19
query19	1.94	1.81	1.79
query20	0.01	0.01	0.01
query21	15.41	0.93	0.54
query22	0.75	1.22	1.13
query23	14.73	1.40	0.67
query24	6.43	2.10	0.59
query25	0.49	0.18	0.14
query26	0.69	0.16	0.15
query27	0.05	0.06	0.05
query28	9.41	0.88	0.42
query29	12.56	3.88	3.24
query30	0.25	0.10	0.07
query31	2.83	0.60	0.38
query32	3.22	0.55	0.48
query33	3.09	3.08	3.00
query34	15.74	5.07	4.51
query35	4.52	4.49	4.49
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 102.52 s
Total hot run time: 29.42 s

doris-robot · 2025-05-09T03:57:13Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.69% (14897/26752)
Line Coverage	44.49% (131374/295302)
Region Coverage	43.39% (66195/152545)
Branch Coverage	38.07% (33805/88796)

hello-stephen · 2025-05-12T03:36:28Z

BE Regression && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.40% (20842/26249)
Line Coverage	72.62% (214085/294800)
Region Coverage	70.79% (126641/178905)
Branch Coverage	64.45% (65307/101328)

dataroaring

LGTM

github-actions · 2025-05-12T13:42:28Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-05-12T13:42:31Z

PR approved by anyone and no changes requested.

#50844) Cherry-picked from #50714 Co-authored-by: walter <maochuan@selectdb.com>

[chore](binlog) allow reading binlogs from follower

6a7f84d

dataroaring reviewed May 8, 2025

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java Outdated Show resolved Hide resolved

fixup

5a4ded2

dataroaring approved these changes May 12, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 12, 2025

github-actions bot added the reviewed label May 12, 2025

bobhan1 approved these changes May 12, 2025

View reviewed changes

dataroaring added the dev/3.0.x label May 13, 2025

w41ter merged commit b2c4641 into apache:master May 13, 2025
24 of 27 checks passed

w41ter deleted the allow_reading_binlogs_from_follower branch May 13, 2025 07:11

github-actions bot pushed a commit that referenced this pull request May 13, 2025

[chore](binlog) allow reading binlogs from follower (#50714)

7a3b775

github-actions bot mentioned this pull request May 13, 2025

branch-3.0: [chore](binlog) allow reading binlogs from follower #50714 #50844

Merged

dataroaring pushed a commit that referenced this pull request May 13, 2025

branch-3.0: [chore](binlog) allow reading binlogs from follower #50714 (

60d5073

#50844) Cherry-picked from #50714 Co-authored-by: walter <maochuan@selectdb.com>

dataroaring added dev/3.0.6-merged and removed dev/3.0.x labels May 13, 2025

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025

[chore](binlog) allow reading binlogs from follower (apache#50714)

0b892b1

[chore](binlog) allow reading binlogs from follower #50714

[chore](binlog) allow reading binlogs from follower #50714

Uh oh!

Conversation

w41ter commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented May 8, 2025

Uh oh!

w41ter commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

hello-stephen commented May 8, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 8, 2025

BE Regression && UT Coverage Report

Uh oh!

w41ter commented May 9, 2025

Uh oh!

doris-robot commented May 9, 2025

Uh oh!

doris-robot commented May 9, 2025

Uh oh!

doris-robot commented May 9, 2025

Uh oh!

doris-robot commented May 9, 2025

Uh oh!

doris-robot commented May 9, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 12, 2025

BE Regression && UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 12, 2025

Uh oh!

github-actions bot commented May 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

w41ter commented May 8, 2025 •

edited

Loading