Skip to content

branch-3.0: [feat](oss) support oss-hdfs root policy #50678 #50960

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
May 17, 2025
Merged

branch-3.0: [feat](oss) support oss-hdfs root policy #50678 #50960

dataroaring merged 1 commit into from
May 17, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented May 15, 2025

Cherry-picked from #50678

@morningman
[feat](oss) support oss-hdfs root policy (#50678)
a453d37 
### What problem does this PR solve?

The root policy is a feature for aliyun OSS-HDFS, it will mapping a hdfs
path to the oss path,
eg:
`hdfs://my_root_policy_name/` ->
`oss://emr-dev-oss.cn-beijing.oss-dls.aliyuncs.com/`

So when root policy is enabled, the location of a hive table will be
`hdfs://my_root_policy_name/xxx`
instead of `oss://emr-dev-oss.cn-beijing.oss-dls.aliyuncs.com/xxx`

This PR covers this case by adding a new catalog property
`oss.root_policy`.
Default is false, if set to true, the location like
`hdfs://my_root_policy_name/` will be treated separately
to fit the jindofs format.
@github-actions github-actions bot requested a review from dataroaring as a code owner May 15, 2025 15:36
@hello-stephen
Copy link
Contributor

hello-stephen commented May 15, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this May 15, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented May 15, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 15, 2025

TPC-H: Total hot run time: 39726 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a453d370de136d6e8c92095a29028b58ee6287f4, data reload: false

------ Round 1 ----------------------------------
q1	17586	6670	6571	6571
q2	2058	167	185	167
q3	10546	1072	1114	1072
q4	10590	793	772	772
q5	7754	2853	2732	2732
q6	221	138	134	134
q7	960	627	603	603
q8	9342	1922	2042	1922
q9	6630	6369	6446	6369
q10	7001	2225	2283	2225
q11	467	269	259	259
q12	392	217	216	216
q13	17779	2949	2986	2949
q14	237	205	228	205
q15	506	463	459	459
q16	687	580	579	579
q17	960	515	566	515
q18	7258	6638	6636	6636
q19	1403	1078	1039	1039
q20	469	204	197	197
q21	3976	3229	3108	3108
q22	1098	997	1002	997
Total cold run time: 107920 ms
Total hot run time: 39726 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6555	6541	6562	6541
q2	324	238	242	238
q3	2876	2819	2792	2792
q4	2045	1780	1754	1754
q5	5783	5736	5733	5733
q6	207	128	132	128
q7	2260	1825	1810	1810
q8	3365	3541	3533	3533
q9	8958	8801	8941	8801
q10	3529	3495	3496	3495
q11	586	509	492	492
q12	792	597	611	597
q13	10227	3214	3114	3114
q14	304	277	282	277
q15	520	464	467	464
q16	690	674	642	642
q17	1823	1606	1590	1590
q18	8334	7801	7749	7749
q19	1638	1580	1471	1471
q20	2039	1855	1855	1855
q21	5469	5176	5356	5176
q22	1126	1057	1073	1057
Total cold run time: 69450 ms
Total hot run time: 59309 ms

@doris-robot
Copy link

doris-robot commented May 15, 2025

TPC-DS: Total hot run time: 198052 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a453d370de136d6e8c92095a29028b58ee6287f4, data reload: false

query1	1333	916	899	899
query2	6233	2024	1998	1998
query3	10837	4368	4259	4259
query4	61204	29043	23453	23453
query5	5180	485	459	459
query6	419	187	183	183
query7	5475	324	310	310
query8	316	233	233	233
query9	8387	2649	2625	2625
query10	461	283	271	271
query11	17426	15312	15731	15312
query12	158	106	110	106
query13	1459	455	435	435
query14	10038	7561	7373	7373
query15	207	181	190	181
query16	7071	433	539	433
query17	1151	612	583	583
query18	1795	328	332	328
query19	232	177	165	165
query20	121	108	113	108
query21	211	107	111	107
query22	4512	4365	4555	4365
query23	35015	34340	34682	34340
query24	6347	3015	3047	3015
query25	546	447	415	415
query26	668	183	184	183
query27	1812	357	360	357
query28	3750	2502	2434	2434
query29	723	503	442	442
query30	264	167	163	163
query31	990	847	842	842
query32	65	59	54	54
query33	436	298	289	289
query34	953	524	535	524
query35	867	738	723	723
query36	1082	946	974	946
query37	124	70	67	67
query38	4095	4065	3946	3946
query39	1522	1466	1475	1466
query40	208	111	114	111
query41	53	54	49	49
query42	119	103	107	103
query43	565	514	493	493
query44	1243	823	849	823
query45	189	166	168	166
query46	1185	762	773	762
query47	2042	1890	1882	1882
query48	497	413	433	413
query49	728	388	401	388
query50	898	442	446	442
query51	7397	7313	7168	7168
query52	104	96	95	95
query53	274	198	190	190
query54	592	484	469	469
query55	84	77	80	77
query56	274	270	250	250
query57	1276	1125	1157	1125
query58	265	223	228	223
query59	3222	2984	3072	2984
query60	287	260	292	260
query61	115	112	109	109
query62	800	678	701	678
query63	223	200	199	199
query64	1431	694	690	690
query65	3310	3234	3254	3234
query66	721	330	296	296
query67	15892	15501	15624	15501
query68	4288	592	577	577
query69	439	268	268	268
query70	1155	1134	1125	1125
query71	365	264	265	264
query72	6324	3775	4015	3775
query73	793	349	375	349
query74	9473	9001	9064	9001
query75	3351	2677	2693	2677
query76	2015	1130	1037	1037
query77	519	292	278	278
query78	10532	9657	9593	9593
query79	1316	611	614	611
query80	899	455	433	433
query81	526	245	246	245
query82	1265	94	95	94
query83	240	148	143	143
query84	282	85	74	74
query85	889	308	293	293
query86	341	269	291	269
query87	4400	4269	4260	4260
query88	3637	2438	2406	2406
query89	425	300	302	300
query90	2002	193	186	186
query91	187	173	152	152
query92	69	50	54	50
query93	1801	551	562	551
query94	754	289	304	289
query95	361	262	256	256
query96	646	294	293	293
query97	3362	3146	3149	3146
query98	219	205	198	198
query99	1558	1289	1287	1287
Total cold run time: 311912 ms
Total hot run time: 198052 ms

@doris-robot
Copy link

doris-robot commented May 15, 2025

ClickBench: Total hot run time: 32.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a453d370de136d6e8c92095a29028b58ee6287f4, data reload: false

query1	0.04	0.03	0.02
query2	0.08	0.03	0.04
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.54	0.50	0.53
query6	1.14	0.73	0.74
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.59	0.50	0.48
query10	0.53	0.54	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	2.86	2.88	2.74
query15	0.91	0.83	0.83
query16	0.39	0.39	0.37
query17	1.07	1.06	1.02
query18	0.24	0.22	0.23
query19	1.96	1.78	1.99
query20	0.01	0.01	0.01
query21	15.36	0.60	0.58
query22	2.70	2.89	1.86
query23	17.06	1.01	0.82
query24	2.93	1.39	1.00
query25	0.29	0.13	0.06
query26	0.52	0.13	0.13
query27	0.05	0.05	0.06
query28	10.36	0.48	0.49
query29	12.58	3.30	3.21
query30	0.25	0.06	0.06
query31	2.87	0.40	0.39
query32	3.23	0.47	0.46
query33	2.97	3.01	3.06
query34	17.18	4.44	4.46
query35	4.60	4.50	4.55
query36	0.67	0.49	0.47
query37	0.09	0.06	0.07
query38	0.05	0.03	0.04
query39	0.04	0.03	0.02
query40	0.17	0.12	0.13
query41	0.07	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.3 s
Total hot run time: 32.12 s

dataroaring
dataroaring approved these changes May 17, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 62a4bdb into branch-3.0 May 17, 2025
23 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-50678-branch-3.0 branch May 17, 2025 03:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hello-stephen @doris-robot @dataroaring @morningman