Skip to content

[fix](broker-load) Support token authentication for S3 temporary credentials in broker-load#57147

Closed
0AyanamiRei wants to merge 2 commits intoapache:branch-3.0from
0AyanamiRei:fix18068
Closed

[fix](broker-load) Support token authentication for S3 temporary credentials in broker-load#57147
0AyanamiRei wants to merge 2 commits intoapache:branch-3.0from
0AyanamiRei:fix18068

Conversation

@0AyanamiRei
Copy link
Copy Markdown
Contributor

@0AyanamiRei 0AyanamiRei commented Oct 20, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

We did not correctly pass the TOKEN parameter in the broker properties.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Oct 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@0AyanamiRei
Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 40100 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45093c9aed7615d32503beb2a2393d942c8cf6a6, data reload: false

------ Round 1 ----------------------------------
q1	17586	6937	6660	6660
q2	2049	199	186	186
q3	10636	1106	1168	1106
q4	10517	768	689	689
q5	7753	2881	2807	2807
q6	217	133	134	133
q7	971	629	613	613
q8	9564	1960	2022	1960
q9	8024	6395	6385	6385
q10	7015	2273	2311	2273
q11	448	265	265	265
q12	407	227	222	222
q13	17783	2995	3000	2995
q14	242	212	209	209
q15	511	469	458	458
q16	496	384	379	379
q17	973	568	564	564
q18	7313	6720	6700	6700
q19	1433	1019	1022	1019
q20	471	200	208	200
q21	3918	3287	3362	3287
q22	1109	996	990	990
Total cold run time: 109436 ms
Total hot run time: 40100 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6634	6593	6548	6548
q2	326	227	232	227
q3	3089	2946	2958	2946
q4	2055	1866	1792	1792
q5	5734	5735	5713	5713
q6	210	132	135	132
q7	2219	1876	1832	1832
q8	3429	3635	3560	3560
q9	8861	8968	8976	8968
q10	3583	3504	3533	3504
q11	595	498	499	498
q12	829	592	662	592
q13	8859	3135	3196	3135
q14	312	270	285	270
q15	507	463	459	459
q16	504	459	440	440
q17	1890	1684	1625	1625
q18	8206	7815	7750	7750
q19	1662	1500	1540	1500
q20	2045	1886	1869	1869
q21	5298	5233	5088	5088
q22	1155	1060	1024	1024
Total cold run time: 68002 ms
Total hot run time: 59472 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 194313 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 45093c9aed7615d32503beb2a2393d942c8cf6a6, data reload: false

query1	940	416	417	416
query2	6231	1945	1839	1839
query3	8707	199	197	197
query4	34011	24122	24052	24052
query5	3596	470	437	437
query6	282	188	180	180
query7	4209	314	319	314
query8	283	227	237	227
query9	9380	2558	2552	2552
query10	467	266	263	263
query11	18064	15790	15646	15646
query12	160	103	102	102
query13	1555	423	430	423
query14	8960	7035	7681	7035
query15	245	180	183	180
query16	8083	502	479	479
query17	1584	606	613	606
query18	2224	349	337	337
query19	293	168	173	168
query20	123	118	123	118
query21	222	114	114	114
query22	4652	4506	4414	4414
query23	35577	34087	33933	33933
query24	11454	2976	3017	2976
query25	735	441	436	436
query26	1789	185	182	182
query27	2990	365	381	365
query28	7886	2171	2162	2162
query29	1084	474	483	474
query30	291	172	183	172
query31	1026	824	848	824
query32	101	69	65	65
query33	806	310	308	308
query34	1039	536	529	529
query35	885	792	740	740
query36	1098	949	929	929
query37	183	69	65	65
query38	4135	3958	3952	3952
query39	1549	1479	1498	1479
query40	264	102	112	102
query41	52	50	48	48
query42	110	104	107	104
query43	530	498	478	478
query44	1271	810	811	810
query45	187	167	172	167
query46	1200	761	741	741
query47	2026	1953	1964	1953
query48	492	387	379	379
query49	1051	415	418	415
query50	844	440	446	440
query51	7531	7457	7322	7322
query52	103	96	94	94
query53	263	184	190	184
query54	1288	480	477	477
query55	80	77	81	77
query56	271	258	245	245
query57	1345	1218	1199	1199
query58	230	222	213	213
query59	3238	3085	2983	2983
query60	295	277	269	269
query61	113	112	113	112
query62	865	698	706	698
query63	235	190	198	190
query64	4909	660	657	657
query65	3541	3394	3382	3382
query66	1449	314	297	297
query67	15933	15573	15528	15528
query68	4829	579	583	579
query69	455	270	263	263
query70	1194	1124	1084	1084
query71	347	294	260	260
query72	6208	4166	4195	4166
query73	768	348	347	347
query74	10716	9285	8964	8964
query75	3479	2679	2691	2679
query76	2706	1061	1187	1061
query77	384	288	275	275
query78	11140	9851	9872	9851
query79	2286	610	598	598
query80	1099	441	434	434
query81	561	231	230	230
query82	612	95	91	91
query83	234	145	146	145
query84	228	82	77	77
query85	1382	308	302	302
query86	455	294	291	291
query87	4439	4242	4275	4242
query88	3816	2402	2380	2380
query89	422	297	294	294
query90	1938	188	188	188
query91	181	153	151	151
query92	61	51	50	50
query93	2479	548	554	548
query94	834	301	303	301
query95	366	258	259	258
query96	627	288	286	286
query97	3407	3181	3155	3155
query98	226	205	202	202
query99	1547	1307	1314	1307
Total cold run time: 306041 ms
Total hot run time: 194313 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 30.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 45093c9aed7615d32503beb2a2393d942c8cf6a6, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.11	0.10
query5	0.53	0.52	0.52
query6	1.12	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.57	0.51	0.51
query10	0.57	0.55	0.55
query11	0.15	0.10	0.10
query12	0.14	0.10	0.11
query13	0.61	0.59	0.59
query14	0.79	0.81	0.79
query15	0.84	0.82	0.83
query16	0.39	0.39	0.41
query17	1.02	1.00	1.07
query18	0.24	0.23	0.23
query19	1.97	1.76	1.75
query20	0.01	0.01	0.01
query21	15.39	0.59	0.57
query22	2.38	1.78	1.88
query23	16.92	1.02	0.92
query24	3.27	0.98	1.08
query25	0.36	0.22	0.09
query26	0.28	0.14	0.13
query27	0.04	0.04	0.04
query28	10.23	0.46	0.49
query29	12.63	3.27	3.25
query30	0.25	0.06	0.06
query31	2.85	0.39	0.38
query32	3.26	0.46	0.45
query33	2.98	3.02	3.05
query34	17.13	4.58	4.54
query35	4.52	4.51	4.60
query36	0.67	0.51	0.50
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.02	0.03
query40	0.16	0.13	0.13
query41	0.09	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.63 s
Total hot run time: 30.36 s

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/4) 🎉
Increment coverage report
Complete coverage report

@0AyanamiRei
Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 40483 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 718944515f1210316fffdf76bafba8fe0664e9b8, data reload: false

------ Round 1 ----------------------------------
q1	17633	7334	6739	6739
q2	2053	199	176	176
q3	10697	1176	1244	1176
q4	10509	738	694	694
q5	7767	2953	2827	2827
q6	219	135	135	135
q7	999	645	636	636
q8	9356	1958	2041	1958
q9	6741	6422	6418	6418
q10	6979	2314	2263	2263
q11	467	285	283	283
q12	422	226	227	226
q13	17791	3074	3028	3028
q14	254	206	222	206
q15	548	463	458	458
q16	476	385	382	382
q17	1037	577	625	577
q18	7382	6718	6797	6718
q19	1414	1031	1063	1031
q20	476	205	209	205
q21	4400	3326	3349	3326
q22	1124	1021	1043	1021
Total cold run time: 108744 ms
Total hot run time: 40483 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6620	6586	6543	6543
q2	327	236	231	231
q3	3164	2926	2919	2919
q4	2136	1829	1864	1829
q5	5804	5751	5759	5751
q6	210	132	134	132
q7	2250	1799	1764	1764
q8	3377	3605	3448	3448
q9	8880	8913	8940	8913
q10	3595	3521	3600	3521
q11	599	495	514	495
q12	811	622	615	615
q13	9037	3199	3225	3199
q14	307	274	268	268
q15	519	455	482	455
q16	519	454	436	436
q17	1892	1619	1657	1619
q18	8466	7899	7809	7809
q19	1693	1590	1572	1572
q20	2146	1875	1897	1875
q21	5267	4955	5148	4955
q22	1184	1086	1062	1062
Total cold run time: 68803 ms
Total hot run time: 59411 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 196510 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 718944515f1210316fffdf76bafba8fe0664e9b8, data reload: false

query1	950	423	391	391
query2	6236	2027	1990	1990
query3	8711	217	217	217
query4	33604	23790	23551	23551
query5	3515	508	514	508
query6	298	203	192	192
query7	4229	348	343	343
query8	314	238	234	234
query9	9779	2819	2820	2819
query10	498	280	273	273
query11	18018	15328	15583	15328
query12	175	108	110	108
query13	1569	461	465	461
query14	10219	6860	6911	6860
query15	263	177	191	177
query16	8016	514	494	494
query17	1594	633	612	612
query18	2146	347	351	347
query19	263	171	169	169
query20	128	117	123	117
query21	217	115	114	114
query22	4969	4685	4577	4577
query23	35276	34454	33980	33980
query24	12265	3052	3058	3052
query25	682	433	432	432
query26	1717	191	179	179
query27	3022	375	396	375
query28	8221	2418	2391	2391
query29	1067	484	462	462
query30	270	177	171	171
query31	1066	826	845	826
query32	102	62	64	62
query33	769	332	337	332
query34	975	569	560	560
query35	878	771	754	754
query36	1150	1013	1041	1013
query37	282	73	73	73
query38	4128	3975	4089	3975
query39	1608	1476	1544	1476
query40	258	107	110	107
query41	53	51	54	51
query42	126	111	111	111
query43	572	538	531	531
query44	1460	905	907	905
query45	197	174	177	174
query46	1193	772	816	772
query47	2070	1973	1985	1973
query48	510	435	413	413
query49	1190	427	438	427
query50	876	478	465	465
query51	7410	7534	7466	7466
query52	111	101	96	96
query53	287	203	201	201
query54	1301	513	520	513
query55	86	87	85	85
query56	295	266	268	266
query57	1348	1236	1259	1236
query58	250	224	230	224
query59	3421	3198	3216	3198
query60	311	284	277	277
query61	117	120	115	115
query62	869	698	733	698
query63	235	214	214	214
query64	4945	734	807	734
query65	3411	3391	3318	3318
query66	1570	328	352	328
query67	16590	16015	15806	15806
query68	5406	633	647	633
query69	454	305	303	303
query70	1220	1203	1188	1188
query71	362	278	286	278
query72	6346	4299	4209	4209
query73	822	395	400	395
query74	10703	9087	8985	8985
query75	3392	2669	2701	2669
query76	3283	1070	1137	1070
query77	395	304	292	292
query78	10793	9742	9722	9722
query79	2414	641	681	641
query80	1074	467	461	461
query81	550	221	228	221
query82	568	99	94	94
query83	232	155	157	155
query84	242	86	85	85
query85	1560	324	319	319
query86	459	302	316	302
query87	4476	4251	4297	4251
query88	4486	2638	2608	2608
query89	411	312	306	306
query90	2090	194	192	192
query91	196	159	162	159
query92	72	53	54	53
query93	2164	624	629	624
query94	933	313	325	313
query95	379	277	275	275
query96	668	303	306	303
query97	3371	3191	3206	3191
query98	227	207	207	207
query99	1513	1307	1337	1307
Total cold run time: 311512 ms
Total hot run time: 196510 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 30.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 718944515f1210316fffdf76bafba8fe0664e9b8, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.06	0.08
query4	1.61	0.10	0.11
query5	0.54	0.53	0.55
query6	1.13	0.74	0.72
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.57	0.50	0.51
query10	0.56	0.56	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.11
query13	0.62	0.60	0.60
query14	0.80	0.84	0.81
query15	0.86	0.85	0.85
query16	0.41	0.40	0.39
query17	1.05	1.01	1.09
query18	0.26	0.24	0.24
query19	1.91	1.81	1.88
query20	0.02	0.02	0.02
query21	15.36	0.62	0.61
query22	2.39	2.12	1.28
query23	17.08	0.95	0.97
query24	2.86	1.76	1.01
query25	0.28	0.10	0.05
query26	0.47	0.16	0.14
query27	0.05	0.05	0.04
query28	9.83	0.56	0.49
query29	12.57	3.39	3.38
query30	0.25	0.06	0.07
query31	2.83	0.40	0.40
query32	3.23	0.48	0.47
query33	3.02	3.08	3.14
query34	17.14	4.55	4.52
query35	4.60	4.55	4.56
query36	0.69	0.48	0.51
query37	0.10	0.07	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.13	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.05	0.04	0.04
Total cold run time: 104.23 s
Total hot run time: 30.36 s

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/6) 🎉
Increment coverage report
Complete coverage report

@0AyanamiRei
Copy link
Copy Markdown
Contributor Author

not mr in here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants