Skip to content

branch-3.1: [chore](param-refactor)Support independent AWS connection timeout settings for each object storage #54882 #54980

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
CalvinKirs merged 1 commit into from
Aug 19, 2025
Merged

branch-3.1: [chore](param-refactor)Support independent AWS connection timeout settings for each object storage #54882 #54980

CalvinKirs merged 1 commit into from
Aug 19, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Aug 19, 2025

Cherry-picked from #54882

@CalvinKirs
[chore](param-refactor)Support independent AWS connection timeout set…
2943ea5 
…tings for each object storage (#54882)

…

### What problem does this PR solve?

Previously, connector properties such as use_path_style,
connection.timeout, and request.timeout were shared across all object
storage connectors.
When multiple object storages (e.g., OSS, OBS, S3) were configured,
their connection-related parameters could overwrite each other due to
identical property names.


After:
Now, each object storage type supports prefixed property names to
isolate configurations.
For example:


```
# OSS-specific settings
oss.connection.timeout=8000
oss.connection.request.timeout=15000
oss.use_path_style=true

# OBS-specific settings
obs.connection.timeout=12000
obs.connection.request.timeout=20000
obs.use_path_style=false
```
@github-actions github-actions bot requested a review from morrySnow as a code owner August 19, 2025 03:29
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Aug 19, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 19, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

TPC-H: Total hot run time: 32712 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2943ea5dadbe395da0dd1f13292eb19350583b11, data reload: false

------ Round 1 ----------------------------------
q1	17585	5549	5482	5482
q2	2025	410	289	289
q3	11876	1225	737	737
q4	10299	878	452	452
q5	9037	2387	2131	2131
q6	187	164	132	132
q7	883	743	607	607
q8	9333	1447	1153	1153
q9	5222	4967	4897	4897
q10	6764	2286	1825	1825
q11	465	281	262	262
q12	339	358	211	211
q13	17764	3754	3020	3020
q14	224	224	204	204
q15	521	493	463	463
q16	427	439	384	384
q17	607	867	369	369
q18	6615	6350	6369	6350
q19	1083	957	547	547
q20	331	348	214	214
q21	2964	2205	1990	1990
q22	1024	1036	993	993
Total cold run time: 105575 ms
Total hot run time: 32712 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5517	5506	5532	5506
q2	237	324	232	232
q3	2244	2645	2317	2317
q4	1344	1772	1366	1366
q5	4406	4965	4974	4965
q6	169	161	131	131
q7	2101	1952	1860	1860
q8	2615	2840	2704	2704
q9	7131	7160	7201	7160
q10	3047	3321	2757	2757
q11	574	503	504	503
q12	692	758	596	596
q13	3356	3795	3125	3125
q14	277	298	277	277
q15	512	473	467	467
q16	426	478	429	429
q17	1251	1756	1258	1258
q18	7620	7378	7344	7344
q19	827	1132	1099	1099
q20	2018	2046	1900	1900
q21	5335	4814	4577	4577
q22	1084	1064	999	999
Total cold run time: 52783 ms
Total hot run time: 51572 ms

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

TPC-DS: Total hot run time: 193222 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2943ea5dadbe395da0dd1f13292eb19350583b11, data reload: false

query1	988	393	401	393
query2	6230	2045	1931	1931
query3	8683	196	192	192
query4	33796	23811	23911	23811
query5	3691	591	456	456
query6	278	189	200	189
query7	4192	501	314	314
query8	309	259	251	251
query9	9429	2618	2595	2595
query10	508	349	262	262
query11	18120	15381	15303	15303
query12	163	109	106	106
query13	1545	549	424	424
query14	9468	6629	6916	6629
query15	209	194	184	184
query16	7892	691	492	492
query17	1593	777	610	610
query18	2023	444	334	334
query19	210	201	167	167
query20	123	121	114	114
query21	210	128	107	107
query22	4474	4617	4387	4387
query23	35905	34270	34684	34270
query24	8938	2742	2717	2717
query25	563	498	447	447
query26	1165	292	188	188
query27	2678	496	382	382
query28	5584	2247	2218	2218
query29	813	612	465	465
query30	243	220	178	178
query31	1038	954	833	833
query32	88	55	56	55
query33	504	378	301	301
query34	740	845	503	503
query35	774	788	765	765
query36	1031	1073	987	987
query37	120	104	68	68
query38	4033	4068	4032	4032
query39	1543	1481	1498	1481
query40	208	121	107	107
query41	51	71	50	50
query42	128	108	106	106
query43	512	551	501	501
query44	1341	839	827	827
query45	184	186	178	178
query46	905	1072	678	678
query47	1961	1960	1895	1895
query48	416	436	352	352
query49	760	496	416	416
query50	675	675	433	433
query51	7378	7256	7264	7256
query52	105	102	97	97
query53	232	256	196	196
query54	552	559	491	491
query55	84	85	78	78
query56	270	276	253	253
query57	1288	1280	1194	1194
query58	230	230	238	230
query59	3148	3254	3111	3111
query60	291	296	276	276
query61	117	118	113	113
query62	802	762	688	688
query63	248	204	200	200
query64	4445	1022	670	670
query65	3390	3361	3350	3350
query66	1014	431	314	314
query67	16170	15748	15660	15660
query68	7880	851	545	545
query69	494	305	266	266
query70	1146	1126	1081	1081
query71	381	298	279	279
query72	5796	3781	3792	3781
query73	644	750	356	356
query74	10439	9408	8917	8917
query75	3206	3121	2707	2707
query76	3054	1151	768	768
query77	525	369	279	279
query78	10441	10485	9736	9736
query79	2684	863	627	627
query80	696	541	445	445
query81	523	259	231	231
query82	376	120	92	92
query83	162	161	146	146
query84	245	110	89	89
query85	793	392	304	304
query86	402	307	298	298
query87	4351	4346	4276	4276
query88	5208	2395	2408	2395
query89	419	333	304	304
query90	1778	192	192	192
query91	142	140	109	109
query92	71	59	54	54
query93	1909	896	549	549
query94	689	420	325	325
query95	339	274	272	272
query96	494	622	286	286
query97	3153	3352	3119	3119
query98	224	214	203	203
query99	1548	1408	1330	1330
Total cold run time: 297011 ms
Total hot run time: 193222 ms

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

ClickBench: Total hot run time: 29.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2943ea5dadbe395da0dd1f13292eb19350583b11, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.23	0.06	0.07
query4	1.62	0.11	0.10
query5	0.53	0.51	0.52
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.03	0.04
query9	0.57	0.50	0.50
query10	0.56	0.54	0.56
query11	0.15	0.11	0.11
query12	0.16	0.11	0.11
query13	0.62	0.60	0.60
query14	0.79	0.77	0.80
query15	0.86	0.83	0.82
query16	0.38	0.39	0.39
query17	1.06	1.08	1.08
query18	0.25	0.21	0.22
query19	1.97	1.79	1.93
query20	0.02	0.02	0.01
query21	15.40	0.94	0.57
query22	0.75	0.78	0.63
query23	15.17	1.46	0.57
query24	2.97	1.30	0.97
query25	0.15	0.18	0.06
query26	0.33	0.15	0.13
query27	0.05	0.06	0.05
query28	13.31	1.03	0.45
query29	12.60	4.02	3.31
query30	0.24	0.09	0.07
query31	2.82	0.58	0.38
query32	3.22	0.55	0.47
query33	3.00	3.02	3.04
query34	16.33	5.29	4.55
query35	4.54	4.57	4.58
query36	0.65	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 103.08 s
Total hot run time: 29.01 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 19, 2025

FE UT Coverage Report

Increment line coverage 68.12% (47/69) 🎉
Increment coverage report
Complete coverage report

@CalvinKirs
Copy link
Member

CalvinKirs commented Aug 19, 2025

run p0

@CalvinKirs CalvinKirs merged commit 30aaeeb into branch-3.1 Aug 19, 2025
23 checks passed
@github-actions github-actions bot deleted the auto-pick-54882-branch-3.1 branch August 19, 2025 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @CalvinKirs @morrySnow @dataroaring