[fix](be) Avoid repeated peer connect failures

[deardeng]

Problem:
Peer reads create an uncached BRPC client for every request. When a peer is unreachable, readers across tablets repeatedly connect to the same address with the default 2-second connect timeout and 10 retries.

Impact:
An unavailable cache peer can repeatedly delay reads before remote-storage fallback and generate unnecessary connection attempts and warning logs.

Fix:

• Add a BE-wide, address-level circuit breaker shared by peer readers.
• Open the circuit after a configurable number of consecutive connection or RPC failures (3 by default) for a configurable cooldown (30 seconds by default).
• Reject reads while the circuit is open, allow one probe after cooldown, and clear the failure state after a successful RPC.
• Make uncached BRPC connect timeout and retry count configurable at the call site, and use a 200 ms timeout with no retries for peer reads.

Test:
Stop the peer test server and verify that four reads issue only three RPC attempts, with the final read rejected by the circuit breaker.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[deardeng]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 16864 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cb1c88315e653f77406d105310c55edb4910c03d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17615	3094	3064	3064
q2	2085	263	248	248
q3	10224	878	551	551
q4	4667	249	199	199
q5	7679	555	394	394
q6	135	114	92	92
q7	527	506	380	380
q8	9290	965	888	888
q9	3490	2385	2388	2385
q10	6515	844	696	696
q11	389	202	184	184
q12	615	260	201	201
q13	18115	1524	1152	1152
q14	156	144	141	141
q15	q16	442	392	364	364
q17	1447	846	819	819
q18	3128	2255	2233	2233
q19	1266	953	794	794
q20	385	283	200	200
q21	5697	1653	1881	1653
q22	336	268	226	226
Total cold run time: 94203 ms
Total hot run time: 16864 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3455	3375	3396	3375
q2	512	389	356	356
q3	2177	2351	2144	2144
q4	1190	1175	883	883
q5	2165	2101	2080	2080
q6	170	121	87	87
q7	1020	906	851	851
q8	1608	1432	1420	1420
q9	3105	3100	3072	3072
q10	1867	1790	1630	1630
q11	368	267	250	250
q12	449	431	337	337
q13	1476	1536	1154	1154
q14	176	171	155	155
q15	q16	399	388	352	352
q17	3829	3276	3218	3218
q18	4819	4421	4700	4421
q19	883	921	880	880
q20	987	972	827	827
q21	3822	3134	3244	3134
q22	404	345	310	310
Total cold run time: 34881 ms
Total hot run time: 30936 ms

[hello-stephen]

TPC-DS: Total hot run time: 81783 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cb1c88315e653f77406d105310c55edb4910c03d, data reload: false

query5	4268	415	340	340
query6	392	142	139	139
query7	4924	435	243	243
query8	295	127	125	125
query9	8708	2884	2878	2878
query10	391	222	188	188
query11	5376	1032	930	930
query12	120	71	72	71
query13	1197	451	332	332
query14	6173	2188	2109	2109
query14_1	1961	1957	1958	1957
query15	172	123	110	110
query16	919	373	347	347
query17	817	461	376	376
query18	2346	331	241	241
query19	168	142	112	112
query20	72	71	70	70
query21	205	100	87	87
query22	5423	5320	5372	5320
query23	6659	6277	6135	6135
query23_1	6005	5978	6071	5978
query24	7330	1113	802	802
query24_1	794	769	800	769
query25	456	274	231	231
query26	1229	224	132	132
query27	2784	414	245	245
query28	4690	1496	1480	1480
query29	912	435	333	333
query30	254	157	129	129
query31	847	397	312	312
query32	150	75	69	69
query33	455	207	162	162
query34	993	896	479	479
query35	400	390	338	338
query36	563	557	552	552
query37	124	78	70	70
query38	1001	852	806	806
query39	510	502	493	493
query39_1	475	452	496	452
query40	202	85	74	74
query41	54	50	50	50
query42	72	70	72	70
query43	240	235	209	209
query44	1020	556	539	539
query45	112	101	96	96
query46	800	842	512	512
query47	737	753	718	718
query48	323	314	233	233
query49	537	243	184	184
query50	749	257	191	191
query51	7996	8072	8114	8072
query52	66	69	59	59
query53	202	204	152	152
query54	225	163	149	149
query55	75	61	53	53
query56	195	159	153	153
query57	750	652	632	632
query58	215	158	162	158
query59	1211	1220	1110	1110
query60	228	187	181	181
query61	142	123	133	123
query62	368	213	172	172
query63	170	140	145	140
query64	2783	694	632	632
query65	1601	1619	1541	1541
query66	1944	263	193	193
query67	9553	9552	9356	9356
query68	2742	1152	749	749
query69	343	232	199	199
query70	677	630	628	628
query71	236	184	156	156
query72	2318	1769	1645	1645
query73	639	595	356	356
query74	1565	1224	1137	1137
query75	1132	1072	946	946
query76	2303	719	529	529
query77	256	252	208	208
query78	3955	3727	3235	3235
query79	1230	825	585	585
query80	1172	318	274	274
query81	485	157	131	131
query82	1191	115	95	95
query83	280	205	192	192
query84	297	118	92	92
query85	844	368	304	304
query86	419	176	167	167
query87	1000	958	890	890
query88	2776	2108	2098	2098
query89	274	195	175	175
query90	1968	130	135	130
query91	131	117	98	98
query92	87	70	69	69
query93	1382	1018	663	663
query94	676	270	241	241
query95	518	246	297	246
query96	814	571	265	265
query97	1065	1035	990	990
query98	142	142	130	130
query99	422	344	307	307
Total cold run time: 175905 ms
Total hot run time: 81783 ms

[hello-stephen]

ClickBench: Total hot run time: 14.59 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cb1c88315e653f77406d105310c55edb4910c03d, data reload: false

query1	0.00	0.01	0.00
query2	0.07	0.04	0.03
query3	0.26	0.11	0.11
query4	1.60	0.10	0.10
query5	0.17	0.17	0.16
query6	1.27	0.70	0.65
query7	0.03	0.00	0.01
query8	0.05	0.03	0.03
query9	0.29	0.21	0.21
query10	0.36	0.35	0.35
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.31	0.32	0.31
query14	0.45	0.45	0.45
query15	0.36	0.35	0.35
query16	0.22	0.20	0.22
query17	0.67	0.65	0.62
query18	0.18	0.15	0.16
query19	1.21	1.20	1.21
query20	0.02	0.01	0.01
query21	15.43	0.16	0.11
query22	5.06	0.04	0.04
query23	16.17	0.24	0.10
query24	2.95	0.30	0.25
query25	0.11	0.04	0.04
query26	0.81	0.16	0.13
query27	0.04	0.03	0.02
query28	3.62	0.56	0.30
query29	12.41	3.21	2.56
query30	0.28	0.12	0.12
query31	2.76	0.37	0.17
query32	3.51	0.33	0.24
query33	1.46	1.41	1.37
query34	15.32	2.19	1.78
query35	1.73	1.72	1.70
query36	0.47	0.30	0.29
query37	0.06	0.04	0.03
query38	0.04	0.03	0.03
query39	0.04	0.02	0.02
query40	0.12	0.08	0.07
query41	0.07	0.02	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 90.35 s
Total hot run time: 14.59 s

[deardeng]

/review

[github-actions]

Overall: the PR addresses the repeated-connect latency case, and the new BRPC defaults preserve existing callers, but the address circuit-breaker state is not correctly integrated with peer-candidate recovery and mutable configuration.

Blocking findings:

1. Circuit-open returns generic THRIFT_RPC_ERROR, so both peer-read paths evict candidates and count all-miss attempts even though no RPC was made; this can remove the candidate before the half-open probe.
2. Transport-successful INTERNAL_ERROR responses clear the address failure state before status validation, so repeated RPC failures do not trip the advertised BE-wide breaker.
3. The process-wide health map has no expiry or bound; permanently dead/retired endpoints accumulate until BE shutdown.
4. Mutable threshold/cooldown changes are not reconciled with stored deadlines; raising the threshold after a circuit opens can turn every post-expiry failure into an immediate probe with no cooldown.

Review checkpoints:

• Goal/test: the connection-refusal path is covered by the new unit test and CI BE UT is green, but cooldown, mutable-config, non-OK-status, and state-cleanup behavior are not covered.
• Scope: the diff is focused, but the caller-side candidate lifecycle must be updated with the new circuit status.
• Concurrency: the bthread mutex protects map transitions and the single half-open probe; no separate lock-order issue was found, but the state transition flaws above remain under concurrent readers.
• Lifecycle/memory: global state survives candidate expiry and is not bounded.
• Configuration: both new fields are mutable, yet existing circuit deadlines and probe state are not adjusted on runtime updates.
• Compatibility: all existing get_new_client_no_cache callers retain the prior 2000 ms connect timeout and 10 retries; only peer reads use 200 ms and zero retries.
• Parallel paths: both peer race and sequential fallback paths share the circuit-open misclassification.
• Data/error safety: reads fall back to S3 rather than corrupting data, but peer failure classification and recovery are incorrect.
• User focus: no additional focus was provided; the complete changed-file review is covered above.

Please address the inline findings and add tests for status classification, cooldown/probe transitions after config updates, and bounded health-state cleanup.

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.14% (34650/45506)
Line Coverage	61.20% (390964/638844)
Region Coverage	57.30% (328059/572534)
Branch Coverage	58.13% (149783/257655)