Skip to content

[fix](oss)Support Aliyun OSS S3-Compatible Public Endpoint Format (eg. s3.oss-cn-xxx.aliyuncs.com) #51894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 19, 2025
Merged

[fix](oss)Support Aliyun OSS S3-Compatible Public Endpoint Format (eg. s3.oss-cn-xxx.aliyuncs.com) #51894

merged 1 commit into from
Jun 19, 2025

Conversation

CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented Jun 18, 2025 •
edited by morningman
Loading

What problem does this PR solve?

#50238

FYI https://help.aliyun.com/zh/oss/developer-reference/use-amazon-s3-sdks-to-access-oss

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33924 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0955fbf532f7f5c421dc16dd7f45c0698a086db, data reload: false

------ Round 1 ----------------------------------
q1	17597	5156	5006	5006
q2	1905	297	194	194
q3	10300	1299	693	693
q4	10212	999	546	546
q5	7513	2458	2330	2330
q6	185	164	132	132
q7	893	744	618	618
q8	9327	1293	1187	1187
q9	6807	5115	5068	5068
q10	6911	2369	1964	1964
q11	492	279	265	265
q12	344	349	215	215
q13	17768	3636	3058	3058
q14	230	229	210	210
q15	556	475	483	475
q16	420	423	388	388
q17	566	849	358	358
q18	7496	7193	7190	7190
q19	1215	946	549	549
q20	325	324	214	214
q21	3733	3161	2324	2324
q22	994	1001	940	940
Total cold run time: 105789 ms
Total hot run time: 33924 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5082	5017	5040	5017
q2	234	321	220	220
q3	2224	2659	2322	2322
q4	1374	1785	1371	1371
q5	4244	4110	4434	4110
q6	210	167	124	124
q7	2056	1986	1774	1774
q8	2588	2593	2638	2593
q9	7178	7119	7187	7119
q10	3115	3250	2797	2797
q11	568	508	496	496
q12	698	806	622	622
q13	3469	3870	3278	3278
q14	272	286	280	280
q15	515	499	482	482
q16	457	492	431	431
q17	1164	1562	1368	1368
q18	7626	7475	7376	7376
q19	798	893	1027	893
q20	1958	2012	1926	1926
q21	4782	4437	4330	4330
q22	1008	1054	962	962
Total cold run time: 51620 ms
Total hot run time: 49891 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186237 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0955fbf532f7f5c421dc16dd7f45c0698a086db, data reload: false

query1	1001	389	400	389
query2	6551	1918	1900	1900
query3	6742	220	219	219
query4	26433	23816	23279	23279
query5	4396	621	449	449
query6	309	219	207	207
query7	4620	498	287	287
query8	263	236	223	223
query9	8609	2654	2653	2653
query10	499	333	274	274
query11	15391	14955	14845	14845
query12	156	107	105	105
query13	1658	508	398	398
query14	9208	6186	6159	6159
query15	195	218	167	167
query16	7270	634	488	488
query17	1214	710	558	558
query18	1974	401	290	290
query19	187	200	156	156
query20	120	114	112	112
query21	216	125	111	111
query22	4018	4341	4095	4095
query23	33983	33128	33089	33089
query24	8464	2365	2396	2365
query25	523	449	400	400
query26	1225	275	153	153
query27	2758	502	347	347
query28	4328	2135	2115	2115
query29	802	588	449	449
query30	284	220	207	207
query31	935	845	783	783
query32	76	68	74	68
query33	574	373	316	316
query34	797	877	531	531
query35	792	822	739	739
query36	937	983	878	878
query37	110	100	78	78
query38	4271	4126	4051	4051
query39	1507	1408	1405	1405
query40	215	130	119	119
query41	71	66	70	66
query42	125	110	113	110
query43	520	531	476	476
query44	1348	883	815	815
query45	174	172	167	167
query46	832	1018	622	622
query47	1774	1830	1750	1750
query48	385	425	315	315
query49	739	469	395	395
query50	636	668	398	398
query51	4060	4218	4066	4066
query52	109	108	107	107
query53	228	251	186	186
query54	578	569	523	523
query55	85	81	82	81
query56	306	296	295	295
query57	1201	1192	1121	1121
query58	263	261	254	254
query59	2715	2805	2700	2700
query60	339	305	301	301
query61	179	122	126	122
query62	785	703	638	638
query63	223	184	186	184
query64	4383	1003	673	673
query65	4257	4143	4198	4143
query66	1154	405	312	312
query67	15779	15763	15532	15532
query68	8393	878	520	520
query69	472	314	264	264
query70	1224	1162	1080	1080
query71	466	319	304	304
query72	5582	4781	4984	4781
query73	732	666	352	352
query74	9132	9103	8670	8670
query75	3976	3197	2708	2708
query76	3693	1173	743	743
query77	779	375	295	295
query78	10136	10199	9504	9504
query79	2121	845	584	584
query80	574	521	487	487
query81	505	269	227	227
query82	442	132	97	97
query83	248	244	232	232
query84	245	115	92	92
query85	818	357	311	311
query86	389	307	263	263
query87	4477	4415	4419	4415
query88	3544	2270	2239	2239
query89	370	314	288	288
query90	1864	205	208	205
query91	140	141	113	113
query92	78	62	60	60
query93	1604	954	588	588
query94	670	407	306	306
query95	378	291	283	283
query96	490	573	285	285
query97	2691	2803	2673	2673
query98	230	217	194	194
query99	1463	1444	1265	1265
Total cold run time: 274785 ms
Total hot run time: 186237 ms

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33671 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bb9cc61e58921f5c51ca4d89fab7c40c57f542f4, data reload: false

------ Round 1 ----------------------------------
q1	17579	5147	4938	4938
q2	1945	275	173	173
q3	10331	1264	720	720
q4	10210	1014	519	519
q5	7511	2266	2347	2266
q6	176	160	130	130
q7	888	773	603	603
q8	9336	1291	1058	1058
q9	6814	5030	5168	5030
q10	6901	2411	1953	1953
q11	491	291	274	274
q12	337	349	213	213
q13	17792	3654	3015	3015
q14	217	227	213	213
q15	553	490	476	476
q16	425	421	383	383
q17	611	860	383	383
q18	7581	7226	7294	7226
q19	3264	959	547	547
q20	323	333	220	220
q21	3853	3208	2382	2382
q22	1052	1006	949	949
Total cold run time: 108190 ms
Total hot run time: 33671 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5538	4994	5000	4994
q2	231	327	223	223
q3	2163	2622	2307	2307
q4	1375	1783	1309	1309
q5	4241	4154	4377	4154
q6	219	173	127	127
q7	1996	1922	1750	1750
q8	2631	2596	2504	2504
q9	7246	7208	7263	7208
q10	3045	3228	2857	2857
q11	583	505	487	487
q12	683	784	639	639
q13	3516	3923	3271	3271
q14	272	298	267	267
q15	520	467	465	465
q16	461	485	445	445
q17	1139	1574	1410	1410
q18	7696	7517	7411	7411
q19	769	796	792	792
q20	2056	2062	1934	1934
q21	4937	4362	4494	4362
q22	1087	1069	1052	1052
Total cold run time: 52404 ms
Total hot run time: 49968 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193163 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bb9cc61e58921f5c51ca4d89fab7c40c57f542f4, data reload: false

query1	1412	1041	1007	1007
query2	6308	1822	1807	1807
query3	11133	4782	4607	4607
query4	55095	25887	23416	23416
query5	5204	613	460	460
query6	333	212	205	205
query7	4892	506	289	289
query8	303	239	223	223
query9	5446	2629	2606	2606
query10	448	330	273	273
query11	15053	15445	14831	14831
query12	158	107	112	107
query13	1054	549	423	423
query14	10183	6420	6375	6375
query15	218	205	187	187
query16	7154	676	444	444
query17	1112	757	595	595
query18	1561	456	317	317
query19	198	196	164	164
query20	122	120	122	120
query21	215	123	111	111
query22	4539	4353	4181	4181
query23	34293	33401	33615	33401
query24	6591	2470	2443	2443
query25	453	482	414	414
query26	726	271	159	159
query27	2323	511	355	355
query28	3432	2156	2159	2156
query29	582	566	443	443
query30	273	227	209	209
query31	889	873	807	807
query32	74	64	64	64
query33	460	378	314	314
query34	807	919	557	557
query35	808	811	818	811
query36	924	1011	942	942
query37	122	105	76	76
query38	4280	4350	4324	4324
query39	1505	1470	1482	1470
query40	215	117	108	108
query41	63	59	60	59
query42	122	108	113	108
query43	519	525	483	483
query44	1368	842	842	842
query45	185	180	169	169
query46	870	1037	659	659
query47	1828	1918	1763	1763
query48	406	434	323	323
query49	661	500	435	435
query50	644	700	412	412
query51	4271	4348	4229	4229
query52	109	114	106	106
query53	224	260	193	193
query54	585	595	507	507
query55	93	82	84	82
query56	309	325	313	313
query57	1200	1233	1154	1154
query58	272	275	279	275
query59	2594	2825	2679	2679
query60	333	331	318	318
query61	133	121	144	121
query62	724	748	715	715
query63	228	194	193	193
query64	1762	1040	664	664
query65	4251	4148	4220	4148
query66	713	392	311	311
query67	15793	15840	15440	15440
query68	6697	898	549	549
query69	533	316	271	271
query70	1191	1140	1126	1126
query71	492	315	297	297
query72	5966	4703	4596	4596
query73	1523	584	355	355
query74	8941	9277	9061	9061
query75	4179	3208	2697	2697
query76	4203	1214	792	792
query77	759	375	298	298
query78	10205	10164	9440	9440
query79	2613	806	597	597
query80	621	575	444	444
query81	483	262	226	226
query82	436	122	98	98
query83	302	257	234	234
query84	286	101	83	83
query85	777	351	315	315
query86	369	303	282	282
query87	4438	4409	4289	4289
query88	3678	2358	2302	2302
query89	413	328	299	299
query90	1819	217	214	214
query91	140	155	113	113
query92	76	61	60	60
query93	2052	952	602	602
query94	669	428	308	308
query95	369	305	287	287
query96	496	580	278	278
query97	2747	2820	2662	2662
query98	239	207	212	207
query99	1454	1398	1279	1279
Total cold run time: 300324 ms
Total hot run time: 193163 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bb9cc61e58921f5c51ca4d89fab7c40c57f542f4, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.61	0.11	0.10
query5	0.45	0.42	0.41
query6	1.17	0.65	0.67
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.57	0.51	0.52
query10	0.56	0.59	0.57
query11	0.16	0.10	0.11
query12	0.15	0.11	0.12
query13	0.64	0.61	0.61
query14	0.81	0.82	0.81
query15	0.91	0.86	0.86
query16	0.38	0.39	0.39
query17	1.08	1.05	1.09
query18	0.23	0.21	0.21
query19	2.00	1.82	1.88
query20	0.01	0.01	0.02
query21	15.42	0.90	0.54
query22	0.73	1.21	0.72
query23	14.84	1.39	0.66
query24	6.80	1.51	0.51
query25	0.44	0.27	0.08
query26	0.64	0.16	0.15
query27	0.05	0.06	0.05
query28	9.10	0.88	0.46
query29	12.58	4.02	3.30
query30	0.25	0.09	0.06
query31	2.84	0.61	0.40
query32	3.24	0.57	0.49
query33	3.12	3.09	3.12
query34	15.93	5.47	4.79
query35	4.86	4.83	4.84
query36	0.69	0.50	0.48
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.14
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.16 s
Total hot run time: 29.32 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 18, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@CalvinKirs CalvinKirs merged commit 99778fb into apache:master Jun 19, 2025
27 of 29 checks passed
@CalvinKirs CalvinKirs deleted the master-s3-oss- branch June 19, 2025 01:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@hubgeter hubgeter hubgeter approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.1.x reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@CalvinKirs @hello-stephen @doris-robot @morningman @hubgeter