Skip to content

[enhance](Azure) Use s3Uri to specify the object's bucket and key for azure in FE#37308

Merged
dataroaring merged 3 commits intoapache:masterfrom
ByteYue:use_s3_uri_for_s3_load_azure
Jul 4, 2024
Merged

[enhance](Azure) Use s3Uri to specify the object's bucket and key for azure in FE#37308
dataroaring merged 3 commits intoapache:masterfrom
ByteYue:use_s3_uri_for_s3_load_azure

Conversation

@ByteYue
Copy link
Copy Markdown
Contributor

@ByteYue ByteYue commented Jul 4, 2024 •
edited
Loading

Previously when using s3 load on azure blob storage, user should specify the s3.bucket property. But actually we can get the bucket information from the data infile uri.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 4, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@ByteYue
Copy link
Copy Markdown
Contributor Author

ByteYue commented Jul 4, 2024

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 4, 2024

TPC-H: Total hot run time: 39566 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa3ac7a7e08e17d2e4fdb6c51c774c28e6ea9632, data reload: false

------ Round 1 ----------------------------------
q1	17629	4315	4268	4268
q2	2014	185	182	182
q3	10461	1225	1154	1154
q4	10185	784	852	784
q5	7485	2633	2495	2495
q6	215	135	135	135
q7	939	592	599	592
q8	9231	2048	2036	2036
q9	8791	6496	6443	6443
q10	8984	3661	3714	3661
q11	444	235	234	234
q12	428	230	228	228
q13	17776	2991	3000	2991
q14	263	216	213	213
q15	525	485	500	485
q16	516	370	368	368
q17	953	677	690	677
q18	7988	7484	7335	7335
q19	3123	1560	1382	1382
q20	662	314	311	311
q21	5062	3247	3959	3247
q22	404	345	352	345
Total cold run time: 114078 ms
Total hot run time: 39566 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4409	4280	4256	4256
q2	381	276	263	263
q3	2992	2794	2863	2794
q4	1975	1726	1753	1726
q5	5604	5650	5505	5505
q6	238	138	133	133
q7	2192	1849	1889	1849
q8	3277	3378	3404	3378
q9	8765	8703	8798	8703
q10	4136	4059	3968	3968
q11	613	499	504	499
q12	843	712	733	712
q13	16342	3384	3352	3352
q14	307	292	287	287
q15	545	510	500	500
q16	484	422	427	422
q17	1811	1527	1494	1494
q18	8264	8066	7787	7787
q19	6084	1718	1584	1584
q20	2216	1856	1849	1849
q21	7202	4845	4978	4845
q22	631	585	578	578
Total cold run time: 79311 ms
Total hot run time: 56484 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 4, 2024

TPC-DS: Total hot run time: 173048 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa3ac7a7e08e17d2e4fdb6c51c774c28e6ea9632, data reload: false

query1	924	384	369	369
query2	6444	2496	2336	2336
query3	6631	203	203	203
query4	19056	17521	17394	17394
query5	3641	469	467	467
query6	263	164	154	154
query7	4586	294	302	294
query8	340	319	308	308
query9	8594	2412	2393	2393
query10	579	320	281	281
query11	10558	10003	9882	9882
query12	117	89	84	84
query13	1654	364	368	364
query14	9563	7229	7612	7229
query15	246	188	197	188
query16	7724	316	315	315
query17	1818	559	529	529
query18	1959	284	278	278
query19	204	154	154	154
query20	89	86	81	81
query21	210	132	128	128
query22	4336	4119	4041	4041
query23	33902	33719	33791	33719
query24	7719	2992	2949	2949
query25	636	411	400	400
query26	729	168	164	164
query27	2218	332	331	331
query28	5498	2156	2149	2149
query29	904	644	679	644
query30	255	154	157	154
query31	1000	786	734	734
query32	99	58	55	55
query33	576	320	320	320
query34	869	483	487	483
query35	747	643	663	643
query36	1120	971	953	953
query37	142	78	83	78
query38	2908	2909	2828	2828
query39	944	857	827	827
query40	208	126	126	126
query41	56	54	54	54
query42	117	100	103	100
query43	590	561	550	550
query44	1048	739	722	722
query45	196	171	168	168
query46	1079	694	724	694
query47	1860	1785	1803	1785
query48	360	310	294	294
query49	943	393	401	393
query50	756	381	385	381
query51	6832	6816	6753	6753
query52	106	92	91	91
query53	353	282	279	279
query54	679	448	429	429
query55	75	71	73	71
query56	288	276	255	255
query57	1111	1049	1050	1049
query58	253	242	239	239
query59	3433	3084	3097	3084
query60	293	281	288	281
query61	96	110	91	91
query62	586	438	432	432
query63	318	282	291	282
query64	8494	2262	1742	1742
query65	3140	3096	3144	3096
query66	742	317	319	317
query67	15697	15031	14724	14724
query68	8625	536	540	536
query69	718	422	330	330
query70	1412	1117	1117	1117
query71	483	289	278	278
query72	8815	5163	5660	5163
query73	2338	325	322	322
query74	5894	5525	5470	5470
query75	5130	2623	2627	2623
query76	4998	952	897	897
query77	767	296	304	296
query78	9784	9037	8986	8986
query79	10311	535	517	517
query80	1217	469	460	460
query81	536	224	217	217
query82	515	106	107	106
query83	323	171	162	162
query84	266	83	83	83
query85	1046	278	310	278
query86	365	302	310	302
query87	3334	3104	3116	3104
query88	4835	2371	2385	2371
query89	539	392	391	391
query90	2130	186	185	185
query91	124	99	100	99
query92	62	48	47	47
query93	7326	522	502	502
query94	1427	211	207	207
query95	401	317	311	311
query96	620	273	266	266
query97	3217	3007	3028	3007
query98	213	203	195	195
query99	1185	848	835	835
Total cold run time: 288467 ms
Total hot run time: 173048 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 4, 2024

ClickBench: Total hot run time: 31.03 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fa3ac7a7e08e17d2e4fdb6c51c774c28e6ea9632, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.22	0.04	0.04
query4	1.68	0.07	0.07
query5	0.49	0.48	0.48
query6	1.13	0.73	0.72
query7	0.02	0.02	0.02
query8	0.06	0.05	0.04
query9	0.55	0.48	0.49
query10	0.54	0.54	0.53
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.58
query14	0.77	0.77	0.76
query15	0.84	0.82	0.81
query16	0.36	0.37	0.36
query17	1.00	0.97	0.97
query18	0.22	0.26	0.24
query19	1.90	1.74	1.83
query20	0.01	0.01	0.01
query21	15.39	0.77	0.66
query22	4.30	6.96	2.23
query23	18.32	1.36	1.29
query24	2.13	0.23	0.22
query25	0.16	0.08	0.08
query26	0.30	0.21	0.22
query27	0.47	0.23	0.22
query28	13.25	1.03	1.00
query29	12.62	3.32	3.31
query30	0.25	0.06	0.05
query31	2.87	0.40	0.40
query32	3.25	0.49	0.47
query33	2.87	2.94	2.96
query34	17.04	4.34	4.38
query35	4.39	4.44	4.46
query36	0.65	0.48	0.47
query37	0.19	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.15	0.13	0.12
query41	0.10	0.04	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.79 s
Total hot run time: 31.03 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 4, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 4, 2024

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jul 4, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit dbd98e0 into apache:master Jul 4, 2024
gavinchou pushed a commit that referenced this pull request Jul 4, 2024
@ByteYue @gavinchou
[enhance](Azure) Use s3Uri to specify the object's bucket and key for…
922c7c7 
… azure in FE (#37308)

Previously when using s3 load on azure blob storage, user should specify
the s3.bucket property. But actually we can get the bucket information
from the data infile uri.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ByteYue @doris-robot @gavinchou @dataroaring