Skip to content

[fix](multicatalog) fix hadoop authenticator not inited for existing hms catalog#38475

Merged
lide-reed merged 1 commit into
apache:masterfrom
Yulei-Yang:fix_hadoopAuthenticator_isnull
Jul 30, 2024
Merged

[fix](multicatalog) fix hadoop authenticator not inited for existing hms catalog#38475
lide-reed merged 1 commit into
apache:masterfrom
Yulei-Yang:fix_hadoopAuthenticator_isnull

Conversation

@Yulei-Yang
Copy link
Copy Markdown
Contributor

@Yulei-Yang Yulei-Yang commented Jul 29, 2024 •
edited
Loading

authenticator for hms type catalog has been refactored to be initted only on creating，but existing catalog are not initted.

Issue:

switch hive;
show databases;
ERROR 1105 (HY000): HMSClientException, msg: java.lang.NullPointerException: Cannot invoke "org.apache.doris.common.security.authentication.HadoopAuthenticator.doAs(java.security.PrivilegedExceptionAction)" because "this.hadoopAuthenticator" is null

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 29, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Yulei-Yang
Copy link
Copy Markdown
Contributor Author

Yulei-Yang commented Jul 29, 2024

run buildall

wsjz
wsjz approved these changes Jul 29, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@wsjz wsjz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jul 29, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 29, 2024

TPC-H: Total hot run time: 43486 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f3c0faa7fd686b167cd8528eb12cb62f7a4c6d95, data reload: false

------ Round 1 ----------------------------------
q1	17601	4439	4433	4433
q2	2021	204	219	204
q3	10436	1465	1416	1416
q4	10167	838	970	838
q5	7642	3145	3187	3145
q6	240	139	144	139
q7	1124	640	615	615
q8	9464	1905	2022	1905
q9	8606	6958	6990	6958
q10	8820	4013	3985	3985
q11	429	259	258	258
q12	450	232	235	232
q13	17768	2943	2955	2943
q14	281	240	246	240
q15	541	488	492	488
q16	552	385	388	385
q17	1004	959	941	941
q18	8160	7406	7380	7380
q19	1437	1349	1347	1347
q20	602	336	343	336
q21	5471	5002	5073	5002
q22	374	305	296	296
Total cold run time: 113190 ms
Total hot run time: 43486 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4402	4368	4423	4368
q2	347	227	228	227
q3	3229	3266	3302	3266
q4	2086	2124	2122	2122
q5	5603	5541	5580	5541
q6	250	141	137	137
q7	2249	1803	1808	1803
q8	3646	3670	3647	3647
q9	8637	8806	8677	8677
q10	4137	4065	4174	4065
q11	577	457	478	457
q12	857	597	587	587
q13	16457	3118	3107	3107
q14	298	287	259	259
q15	558	484	476	476
q16	491	417	421	417
q17	1912	1861	1869	1861
q18	8414	7831	7800	7800
q19	3013	1897	1888	1888
q20	2097	1837	1826	1826
q21	6040	5380	5571	5380
q22	571	464	483	464
Total cold run time: 75871 ms
Total hot run time: 58375 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 29, 2024

TPC-DS: Total hot run time: 170844 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f3c0faa7fd686b167cd8528eb12cb62f7a4c6d95, data reload: false

query1	911	388	364	364
query2	6442	1765	1681	1681
query3	6652	212	220	212
query4	20279	17544	17391	17391
query5	3646	526	523	523
query6	289	185	173	173
query7	4619	310	300	300
query8	276	211	192	192
query9	8533	2412	2410	2410
query10	431	286	284	284
query11	10658	10101	10121	10101
query12	118	91	83	83
query13	1629	372	377	372
query14	9281	7141	7633	7141
query15	205	161	167	161
query16	6969	427	416	416
query17	959	550	552	550
query18	1926	277	283	277
query19	189	146	144	144
query20	92	87	88	87
query21	202	101	98	98
query22	4181	4151	3995	3995
query23	33732	33877	33604	33604
query24	9512	3093	3114	3093
query25	672	425	407	407
query26	853	150	148	148
query27	2346	287	296	287
query28	6344	2067	2028	2028
query29	971	453	456	453
query30	238	154	169	154
query31	966	769	772	769
query32	106	59	60	59
query33	687	319	333	319
query34	951	506	510	506
query35	868	763	768	763
query36	1073	891	902	891
query37	169	82	86	82
query38	2980	2799	2779	2779
query39	862	835	829	829
query40	198	122	123	122
query41	48	47	50	47
query42	117	103	108	103
query43	490	430	422	422
query44	1194	737	738	737
query45	215	182	183	182
query46	1105	816	785	785
query47	1833	1685	1694	1685
query48	357	296	290	290
query49	852	430	419	419
query50	922	448	457	448
query51	6692	6755	6669	6669
query52	96	90	90	90
query53	258	183	185	183
query54	606	456	461	456
query55	77	77	78	77
query56	286	263	258	258
query57	1114	1043	1043	1043
query58	255	274	287	274
query59	2506	2601	2447	2447
query60	318	279	287	279
query61	99	97	96	96
query62	894	658	654	654
query63	212	179	190	179
query64	4599	1919	1927	1919
query65	3149	3092	3089	3089
query66	838	342	331	331
query67	15349	14801	14811	14801
query68	4374	585	595	585
query69	501	309	296	296
query70	1129	1055	1104	1055
query71	376	280	274	274
query72	7262	2846	2674	2674
query73	757	335	334	334
query74	6014	5685	5644	5644
query75	3409	2771	2777	2771
query76	2036	1364	1397	1364
query77	440	404	314	314
query78	9548	8880	8942	8880
query79	2721	546	546	546
query80	1208	509	514	509
query81	578	239	225	225
query82	783	136	130	130
query83	245	173	179	173
query84	269	78	81	78
query85	1524	362	315	315
query86	472	301	311	301
query87	3336	3070	3099	3070
query88	3953	2506	2493	2493
query89	392	307	286	286
query90	1762	195	198	195
query91	129	100	103	100
query92	63	48	51	48
query93	2295	635	622	622
query94	772	306	297	297
query95	390	277	275	275
query96	600	286	286	286
query97	3205	3026	3075	3026
query98	222	201	193	193
query99	1621	1284	1274	1274
Total cold run time: 258882 ms
Total hot run time: 170844 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 29, 2024

ClickBench: Total hot run time: 30.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f3c0faa7fd686b167cd8528eb12cb62f7a4c6d95, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.04	0.05
query4	1.67	0.06	0.07
query5	0.47	0.48	0.48
query6	1.13	0.71	0.71
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.57	0.50	0.52
query10	0.56	0.56	0.57
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.61	0.60	0.59
query14	0.77	0.79	0.80
query15	0.91	0.86	0.85
query16	0.35	0.36	0.36
query17	1.02	1.01	1.03
query18	0.22	0.21	0.21
query19	1.88	1.72	1.78
query20	0.01	0.01	0.02
query21	15.42	0.77	0.66
query22	3.62	6.23	2.09
query23	18.00	1.27	1.34
query24	2.12	0.25	0.23
query25	0.18	0.08	0.08
query26	0.33	0.22	0.22
query27	0.45	0.24	0.24
query28	13.17	1.01	0.96
query29	12.53	3.34	3.33
query30	0.28	0.06	0.05
query31	2.86	0.42	0.41
query32	3.24	0.48	0.48
query33	2.94	3.00	3.00
query34	15.47	4.27	4.25
query35	4.31	4.28	4.32
query36	0.68	0.48	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.13
query41	0.10	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.22 s
Total hot run time: 30.86 s

nextdreamblue
nextdreamblue approved these changes Jul 29, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@nextdreamblue nextdreamblue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 29, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jul 29, 2024

PR approved by at least one committer and no changes requested.

lide-reed
lide-reed approved these changes Jul 30, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lide-reed lide-reed merged commit 9832838 into apache:master Jul 30, 2024
@Yulei-Yang Yulei-Yang deleted the fix_hadoopAuthenticator_isnull branch July 30, 2024 03:42
kaka11chen pushed a commit to kaka11chen/doris that referenced this pull request Aug 6, 2024
@Yulei-Yang @kaka11chen
[fix](multicatalog) fix hadoop authenticator not inited for existing …
4ed2d18 
…hms catalog (apache#38475)
@kaka11chen kaka11chen mentioned this pull request Aug 6, 2024
Merged
morningman pushed a commit that referenced this pull request Aug 6, 2024
@kaka11chen @Yulei-Yang
[fix](multicatalog) fix hadoop authenticator not inited for existing …
5066be6 
…hms catalog. (#38930)

Backport #38475.

Co-authored-by: Yulei-Yang <yulei.yang0699@gmail.com>
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@Yulei-Yang @dataroaring
[fix](multicatalog) fix hadoop authenticator not inited for existing …
db6b2cd 
…hms catalog (#38475)
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@Yulei-Yang @dataroaring
[fix](multicatalog) fix hadoop authenticator not inited for existing …
cdd156f 
…hms catalog (#38475)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@Yulei-Yang @HappenLee
[fix](multicatalog) fix hadoop authenticator not inited for existing …
7b6f70e 
…hms catalog (apache#38475)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lide-reed lide-reed lide-reed approved these changes

@nextdreamblue nextdreamblue nextdreamblue approved these changes

+1 more reviewer

@wsjz wsjz wsjz approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Yulei-Yang @doris-robot @lide-reed @wsjz @nextdreamblue @morningman @dataroaring