Skip to content

[feat](param-refactor)Unify FS Factory Creation Logic and Support Broker Properties for Catalog Compatibility #51888

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 19, 2025

Conversation

CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented Jun 18, 2025

What problem does this PR solve?

#50238

Eliminated Redundant bind.broker Usage in Catalog:

Previously, the Catalog used a custom bind.broker field—derived from the user-defined ↳broker.name—to determine whether to use a BrokerFileSystem, and to identify the specific broker.
Since we support many storage backends, keeping this value separately is redundant.
This PR migrates the logic into brokerProperties, allowing the factory to determine everything it needs from a single source of truth.

Refactored FSFactory Interface:

The original FSFactory contained too many scattered methods, making it harder to maintain and extend.
We've consolidated and organized its APIs for better clarity, reducing duplication and improving its usability across modules.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@CalvinKirs
Copy link
Member Author

run buildall

@Thearas
Copy link
Contributor

Thearas commented Jun 18, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34150 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1d673124d83bdc94955a92166b76c43fdb8c2cbc, data reload: false

------ Round 1 ----------------------------------
q1	17611	5146	5072	5072
q2	1910	289	170	170
q3	10328	1239	763	763
q4	10216	1008	528	528
q5	7497	2386	2259	2259
q6	180	163	132	132
q7	905	766	605	605
q8	9323	1305	1080	1080
q9	6716	5119	5070	5070
q10	6944	2379	1954	1954
q11	489	277	280	277
q12	340	349	214	214
q13	17772	3613	3084	3084
q14	231	233	224	224
q15	566	473	474	473
q16	422	434	375	375
q17	582	840	374	374
q18	7618	7502	7298	7298
q19	1895	965	568	568
q20	345	342	224	224
q21	4064	3155	2405	2405
q22	1051	1066	1001	1001
Total cold run time: 107005 ms
Total hot run time: 34150 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5104	5057	5109	5057
q2	250	324	232	232
q3	2171	2617	2340	2340
q4	1371	1783	1322	1322
q5	4233	4128	4354	4128
q6	218	176	133	133
q7	2050	1896	1787	1787
q8	2598	2582	2475	2475
q9	7178	7117	7188	7117
q10	3102	3245	2817	2817
q11	634	523	496	496
q12	708	780	626	626
q13	3455	3889	3283	3283
q14	277	290	270	270
q15	535	487	454	454
q16	439	479	454	454
q17	1135	1436	1399	1399
q18	7835	7480	7534	7480
q19	794	792	842	792
q20	1966	2019	1928	1928
q21	4891	4309	4413	4309
q22	1102	1085	1007	1007
Total cold run time: 52046 ms
Total hot run time: 49906 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193288 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1d673124d83bdc94955a92166b76c43fdb8c2cbc, data reload: false

query1	1394	1040	995	995
query2	6310	1912	1894	1894
query3	11015	4435	4513	4435
query4	54048	24244	23679	23679
query5	5242	509	455	455
query6	376	224	200	200
query7	5037	503	290	290
query8	314	241	215	215
query9	6294	2652	2671	2652
query10	425	322	276	276
query11	15166	15003	14812	14812
query12	162	113	104	104
query13	1108	549	420	420
query14	10314	6360	6464	6360
query15	195	193	187	187
query16	7045	659	476	476
query17	1065	755	588	588
query18	1565	403	309	309
query19	190	193	167	167
query20	123	137	116	116
query21	208	123	129	123
query22	4490	4366	4273	4273
query23	34155	33585	33569	33569
query24	6647	2432	2468	2432
query25	479	494	412	412
query26	729	268	151	151
query27	2441	535	351	351
query28	2988	2177	2168	2168
query29	588	566	429	429
query30	281	231	197	197
query31	869	866	767	767
query32	74	63	62	62
query33	462	377	307	307
query34	809	878	532	532
query35	822	823	799	799
query36	953	988	899	899
query37	121	101	84	84
query38	4225	4314	4360	4314
query39	1514	1423	1441	1423
query40	220	129	115	115
query41	64	62	66	62
query42	166	115	112	112
query43	514	526	510	510
query44	1400	849	848	848
query45	187	173	168	168
query46	880	1026	638	638
query47	1858	1856	1780	1780
query48	403	431	320	320
query49	664	475	409	409
query50	696	682	418	418
query51	4174	4225	4148	4148
query52	120	114	103	103
query53	239	272	197	197
query54	579	589	523	523
query55	86	115	81	81
query56	302	293	319	293
query57	1211	1232	1158	1158
query58	270	267	284	267
query59	2889	2955	2794	2794
query60	337	341	315	315
query61	130	125	126	125
query62	721	747	689	689
query63	234	195	194	194
query64	1816	1026	690	690
query65	4234	4153	4202	4153
query66	788	392	301	301
query67	15837	15609	15461	15461
query68	6382	837	524	524
query69	546	304	279	279
query70	1221	1113	1118	1113
query71	476	334	307	307
query72	5774	4883	4827	4827
query73	1154	711	350	350
query74	9077	9097	8647	8647
query75	3418	3200	2710	2710
query76	3817	1190	751	751
query77	551	381	288	288
query78	10312	10219	9420	9420
query79	2074	826	588	588
query80	618	541	451	451
query81	493	266	235	235
query82	242	130	98	98
query83	259	264	247	247
query84	295	106	82	82
query85	810	352	320	320
query86	409	296	289	289
query87	4427	4470	4371	4371
query88	3713	2290	2287	2287
query89	392	326	278	278
query90	1812	215	214	214
query91	141	140	113	113
query92	74	76	57	57
query93	1813	987	580	580
query94	652	399	306	306
query95	368	291	288	288
query96	505	586	283	283
query97	2733	2723	2647	2647
query98	252	209	213	209
query99	1361	1421	1335	1335
Total cold run time: 297433 ms
Total hot run time: 193288 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.8 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1d673124d83bdc94955a92166b76c43fdb8c2cbc, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.06
query4	1.62	0.10	0.11
query5	0.44	0.41	0.41
query6	1.16	0.66	0.67
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.57	0.53	0.52
query10	0.58	0.58	0.56
query11	0.16	0.11	0.11
query12	0.14	0.11	0.12
query13	0.63	0.61	0.60
query14	0.80	0.83	0.81
query15	0.91	0.87	0.85
query16	0.38	0.39	0.39
query17	1.05	1.04	1.03
query18	0.23	0.21	0.21
query19	1.92	1.80	1.82
query20	0.02	0.01	0.01
query21	15.40	0.92	0.54
query22	0.75	1.19	0.64
query23	14.96	1.38	0.62
query24	8.18	0.64	0.32
query25	0.41	0.17	0.06
query26	0.66	0.16	0.14
query27	0.05	0.05	0.04
query28	8.88	0.89	0.44
query29	12.57	3.98	3.32
query30	0.25	0.09	0.06
query31	2.83	0.60	0.40
query32	3.23	0.56	0.46
query33	3.04	3.10	3.07
query34	16.12	5.35	4.72
query35	4.82	4.81	4.79
query36	0.69	0.50	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.18	0.16	0.14
query41	0.09	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.36 s
Total hot run time: 28.8 s

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33981 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8686d41e49e649d3470f16e6faaf860bb7321560, data reload: false

------ Round 1 ----------------------------------
q1	17635	5280	4994	4994
q2	1933	291	190	190
q3	10291	1307	740	740
q4	10214	997	524	524
q5	7528	2307	2370	2307
q6	185	162	131	131
q7	893	745	631	631
q8	9318	1301	1164	1164
q9	6771	5042	5082	5042
q10	6884	2364	1964	1964
q11	478	298	286	286
q12	347	351	221	221
q13	17775	3695	3067	3067
q14	234	223	212	212
q15	573	478	488	478
q16	423	429	373	373
q17	600	853	380	380
q18	7526	7238	7109	7109
q19	1702	946	567	567
q20	348	346	234	234
q21	3889	3238	2404	2404
q22	1039	986	963	963
Total cold run time: 106586 ms
Total hot run time: 33981 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5219	5021	5004	5004
q2	239	322	217	217
q3	2173	2654	2292	2292
q4	1368	1759	1301	1301
q5	4227	4085	4375	4085
q6	226	173	134	134
q7	1949	1908	1774	1774
q8	2591	2550	2504	2504
q9	7096	7132	7117	7117
q10	3052	3405	2829	2829
q11	579	512	489	489
q12	691	755	626	626
q13	3482	3892	3379	3379
q14	285	292	285	285
q15	513	490	499	490
q16	450	482	447	447
q17	1138	1572	1346	1346
q18	7403	7159	7155	7155
q19	798	809	854	809
q20	1893	1953	1811	1811
q21	4725	4569	4350	4350
q22	1041	1016	981	981
Total cold run time: 51138 ms
Total hot run time: 49425 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183647 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8686d41e49e649d3470f16e6faaf860bb7321560, data reload: false

query1	1002	403	388	388
query2	6536	1800	1803	1800
query3	6759	223	220	220
query4	26027	23705	23447	23447
query5	4335	623	468	468
query6	290	207	194	194
query7	4631	488	290	290
query8	263	228	210	210
query9	8646	2618	2649	2618
query10	500	326	273	273
query11	15285	15040	14746	14746
query12	151	112	112	112
query13	1658	529	419	419
query14	9249	6488	6283	6283
query15	212	195	180	180
query16	7555	625	474	474
query17	1201	731	582	582
query18	2026	425	313	313
query19	192	193	163	163
query20	121	117	123	117
query21	216	125	112	112
query22	4329	4296	4276	4276
query23	33890	33056	32943	32943
query24	8315	2424	2380	2380
query25	524	463	394	394
query26	1217	265	154	154
query27	2720	498	337	337
query28	4289	2119	2094	2094
query29	737	578	431	431
query30	285	227	200	200
query31	947	838	741	741
query32	71	62	61	61
query33	569	365	304	304
query34	782	871	504	504
query35	768	797	749	749
query36	930	968	873	873
query37	108	97	77	77
query38	4097	4214	4011	4011
query39	1465	1429	1410	1410
query40	210	129	107	107
query41	76	70	63	63
query42	128	109	110	109
query43	510	514	479	479
query44	1328	829	844	829
query45	178	175	166	166
query46	836	1026	639	639
query47	1758	1765	1703	1703
query48	380	413	305	305
query49	750	498	426	426
query50	639	669	416	416
query51	4110	4152	4157	4152
query52	114	106	99	99
query53	221	257	189	189
query54	570	577	502	502
query55	91	86	83	83
query56	308	307	296	296
query57	1155	1187	1116	1116
query58	266	265	257	257
query59	2454	2669	2491	2491
query60	332	315	321	315
query61	126	121	152	121
query62	805	732	668	668
query63	221	184	191	184
query64	4243	1002	702	702
query65	4275	4202	4238	4202
query66	1112	409	306	306
query67	15604	15572	15490	15490
query68	8523	881	546	546
query69	473	303	272	272
query70	1201	1113	1041	1041
query71	469	324	286	286
query72	5488	4696	2308	2308
query73	705	583	361	361
query74	9232	9119	8705	8705
query75	3862	3181	2678	2678
query76	3654	1174	769	769
query77	779	373	358	358
query78	10044	10294	9382	9382
query79	1680	784	581	581
query80	576	518	442	442
query81	505	255	226	226
query82	486	126	96	96
query83	243	244	231	231
query84	243	106	92	92
query85	773	343	315	315
query86	340	297	275	275
query87	4380	4492	4305	4305
query88	2896	2331	2360	2331
query89	386	321	292	292
query90	1911	211	207	207
query91	139	145	109	109
query92	72	59	56	56
query93	1047	935	595	595
query94	662	404	301	301
query95	380	286	287	286
query96	488	558	284	284
query97	2780	2799	2675	2675
query98	227	211	210	210
query99	1436	1379	1269	1269
Total cold run time: 271778 ms
Total hot run time: 183647 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 18, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

ClickBench: Total hot run time: 29.56 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8686d41e49e649d3470f16e6faaf860bb7321560, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.11	0.10
query5	0.43	0.41	0.43
query6	1.18	0.67	0.65
query7	0.02	0.02	0.01
query8	0.05	0.03	0.04
query9	0.58	0.51	0.51
query10	0.57	0.58	0.58
query11	0.15	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.61	0.61
query14	0.79	0.82	0.82
query15	0.90	0.88	0.89
query16	0.39	0.39	0.40
query17	1.08	1.02	1.09
query18	0.22	0.21	0.21
query19	1.98	1.81	1.86
query20	0.02	0.01	0.01
query21	15.39	0.90	0.56
query22	0.75	1.10	0.65
query23	15.04	1.36	0.65
query24	6.61	1.14	0.84
query25	0.53	0.11	0.13
query26	0.61	0.17	0.14
query27	0.06	0.06	0.05
query28	9.79	0.93	0.46
query29	12.64	4.02	3.35
query30	0.25	0.09	0.07
query31	2.84	0.61	0.41
query32	3.25	0.57	0.47
query33	3.08	3.08	3.17
query34	16.07	5.36	4.75
query35	4.78	4.85	4.78
query36	0.72	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.02	0.02	0.02
query40	0.17	0.14	0.13
query41	0.09	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.97 s
Total hot run time: 29.56 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 76.32% (29/38) 🎉
Increment coverage report
Complete coverage report

@CalvinKirs CalvinKirs merged commit 3cf4045 into apache:master Jun 19, 2025
25 of 28 checks passed
@CalvinKirs CalvinKirs deleted the master-fs-factore-remote-fs branch June 19, 2025 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.1.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants