Skip to content

Fe obs native sdk spike#63400

Draft
CalvinKirs wants to merge 3 commits into
apache:masterfrom
CalvinKirs:fe-obs-native-sdk-spike
Draft

Fe obs native sdk spike#63400
CalvinKirs wants to merge 3 commits into
apache:masterfrom
CalvinKirs:fe-obs-native-sdk-spike

Conversation

@CalvinKirs
Copy link
Copy Markdown
Member

@CalvinKirs CalvinKirs commented May 19, 2026

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

CalvinKirs added 3 commits May 14, 2026 18:04
@CalvinKirs
[refactor](fe) Add typed filesystem properties for S3
bf360bb 
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Add provider-owned typed filesystem properties and bind S3 runtime paths through S3FileSystemProperties, preparing for S3Properties removal from fe-core.

### Release note

None

### Check List (For Author)

- Test: Unit Test

    - mvn -Dmaven.build.cache.enabled=false -f fe/pom.xml -pl fe-filesystem/fe-filesystem-s3 -am -DfailIfNoTests=false test

    - mvn -Dmaven.build.cache.enabled=false -f fe/pom.xml -pl fe-filesystem/fe-filesystem-oss,fe-filesystem/fe-filesystem-cos,fe-filesystem/fe-filesystem-obs -am -DfailIfNoTests=false test

    - mvn -Dmaven.build.cache.enabled=false -f fe/pom.xml -pl fe-filesystem/fe-filesystem-s3,fe-filesystem/fe-filesystem-oss,fe-filesystem/fe-filesystem-cos,fe-filesystem/fe-filesystem-obs -am -DfailIfNoTests=false test

    - mvn -Dmaven.build.cache.enabled=false -f fe/pom.xml -pl fe-filesystem/fe-filesystem-s3 -am -DskipTests validate

    - env MAVEN_OPTS=-Xmx5g MVN_OPT=-Dmaven.build.cache.enabled=false FE_MAVEN_THREADS=1 FE_MAVEN_RETRY_THREADS=1 CUSTOM_UI_DIST=/mnt/disk1/gq/idea/incubator-doris/ui/dist ./build.sh --fe

- Behavior changed: No

- Does this need documentation: No
@CalvinKirs
[fix](fe) Normalize S3 filesystem defaults
28929d5 
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Normalize S3 endpoint-only configuration in filesystem properties and preserve AWS profile credentials in the default S3 credential provider chain.

### Release note

None

### Check List (For Author)

- Test: Unit Test

    - mvn -f fe/pom.xml -pl fe-filesystem/fe-filesystem-s3 -am -Dmaven.build.cache.enabled=false validate

    - mvn -f fe/pom.xml -pl fe-filesystem/fe-filesystem-s3 -am -Dmaven.build.cache.enabled=false -DfailIfNoTests=false test

- Behavior changed: Yes. S3 endpoint-only filesystem configuration now builds us-east-1 in properties, and DEFAULT credentials include AWS profile credentials before anonymous fallback.

- Does this need documentation: No
@CalvinKirs
[improvement](fe) Add native filesystem SPI for object storage
212b24a 
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Unify FE filesystem SPI capability and typed storage properties so S3-compatible providers can bind parameters consistently while COS, OBS, and OSS can use native SDK-backed file systems.

### Release note

None

### Check List (For Author)

- Test: Unit Test

    - mvn -f fe/pom.xml -pl fe-filesystem/fe-filesystem-cos,fe-filesystem/fe-filesystem-obs -am -Dmaven.build.cache.enabled=false -DfailIfNoTests=false -Dtest=CosFileSystemPropertiesTest,ObsFileSystemPropertiesTest,CosObjStorageTest,ObsObjStorageTest test

    - mvn -f fe/pom.xml -pl fe-filesystem/fe-filesystem-s3,fe-filesystem/fe-filesystem-cos,fe-filesystem/fe-filesystem-obs -am -Dmaven.build.cache.enabled=false -DskipTests package

    - git diff --check

- Behavior changed: No

- Does this need documentation: No
@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 19, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs
Copy link
Copy Markdown
Member Author

CalvinKirs commented May 19, 2026

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 19, 2026

TPC-H: Total hot run time: 31393 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 212b24aa50a212791fa028b77ec38cd6a3d0358a, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17721	3981	3953	3953
q2	q3	10764	1403	818	818
q4	4687	477	342	342
q5	7614	2266	2096	2096
q6	238	178	139	139
q7	937	780	631	631
q8	9426	1683	1621	1621
q9	5231	4891	4917	4891
q10	6404	2083	1803	1803
q11	426	274	252	252
q12	632	414	286	286
q13	18117	3355	2738	2738
q14	267	255	239	239
q15	q16	823	780	707	707
q17	982	906	965	906
q18	6884	5764	5492	5492
q19	1309	1287	1108	1108
q20	538	401	365	365
q21	6241	2850	2633	2633
q22	450	373	442	373
Total cold run time: 99691 ms
Total hot run time: 31393 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4602	4576	4613	4576
q2	q3	4871	5339	4623	4623
q4	2181	2206	1422	1422
q5	4998	4674	4624	4624
q6	239	186	140	140
q7	1857	1684	1509	1509
q8	2399	2021	2081	2021
q9	7643	7231	7194	7194
q10	4471	4402	4003	4003
q11	520	378	368	368
q12	697	721	512	512
q13	3066	3455	2787	2787
q14	271	280	262	262
q15	q16	676	695	610	610
q17	1268	1241	1238	1238
q18	7175	6626	6583	6583
q19	1090	1101	1078	1078
q20	2219	2207	1928	1928
q21	5301	4604	4467	4467
q22	503	448	397	397
Total cold run time: 56047 ms
Total hot run time: 50342 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 19, 2026

TPC-DS: Total hot run time: 169617 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 212b24aa50a212791fa028b77ec38cd6a3d0358a, data reload: false

query5	4352	656	535	535
query6	325	221	207	207
query7	4219	542	320	320
query8	322	228	215	215
query9	8838	3997	3946	3946
query10	463	329	299	299
query11	5662	2393	2215	2215
query12	177	125	123	123
query13	1260	596	443	443
query14	5986	5339	5045	5045
query14_1	4332	4350	4324	4324
query15	208	197	180	180
query16	1007	455	440	440
query17	975	749	610	610
query18	2457	491	366	366
query19	214	215	164	164
query20	140	137	134	134
query21	217	136	125	125
query22	13604	13487	13320	13320
query23	17147	16411	16033	16033
query23_1	16191	16117	16183	16117
query24	7417	1726	1311	1311
query24_1	1315	1303	1308	1303
query25	588	496	445	445
query26	1311	328	173	173
query27	2690	582	342	342
query28	4455	1967	1926	1926
query29	1021	641	524	524
query30	306	243	203	203
query31	1120	1067	939	939
query32	96	83	74	74
query33	543	365	307	307
query34	1160	1138	653	653
query35	773	821	685	685
query36	1395	1362	1227	1227
query37	154	103	91	91
query38	3236	3165	3047	3047
query39	926	940	894	894
query39_1	865	883	895	883
query40	233	156	133	133
query41	74	69	69	69
query42	112	114	109	109
query43	326	328	286	286
query44	
query45	223	218	209	209
query46	1096	1186	719	719
query47	2275	2360	2223	2223
query48	400	409	295	295
query49	648	495	399	399
query50	986	361	259	259
query51	4315	4328	4301	4301
query52	106	109	98	98
query53	260	283	208	208
query54	327	282	268	268
query55	94	92	90	90
query56	326	309	375	309
query57	1438	1372	1332	1332
query58	298	270	261	261
query59	1568	1635	1426	1426
query60	325	325	298	298
query61	153	160	148	148
query62	664	636	550	550
query63	237	201	204	201
query64	2448	803	640	640
query65	
query66	1737	464	349	349
query67	29395	29983	29848	29848
query68	
query69	466	348	297	297
query70	1021	918	992	918
query71	299	275	269	269
query72	2954	2682	2410	2410
query73	805	771	414	414
query74	5059	4946	4755	4755
query75	2670	2583	2290	2290
query76	2269	1100	736	736
query77	393	399	336	336
query78	12181	12371	11666	11666
query79	1444	1027	759	759
query80	664	563	442	442
query81	459	277	239	239
query82	1370	154	124	124
query83	361	276	246	246
query84	297	137	112	112
query85	867	543	434	434
query86	407	361	354	354
query87	3400	3386	3184	3184
query88	3500	2650	2611	2611
query89	436	385	338	338
query90	1967	193	173	173
query91	173	170	141	141
query92	81	77	71	71
query93	1448	1456	921	921
query94	549	349	302	302
query95	671	487	342	342
query96	963	748	349	349
query97	2705	2695	2562	2562
query98	236	232	232	232
query99	1127	1098	990	990
Total cold run time: 251698 ms
Total hot run time: 169617 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 19, 2026

FE Regression Coverage Report

Increment line coverage 0.00% (0/21) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman Awaiting requested review from morningman morningman will be requested when the pull request is marked ready for review morningman is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@CalvinKirs @hello-stephen