[SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . #25201

AngersZhuuuu · 2019-07-19T12:37:08Z

What changes were proposed in this pull request?

#25179
Since origin SparkThriftServer can't proxy client user's authentication about hive and HDFS authentications.
this PR is to enable SparkThriftServer can proxy user's authorities.

For this pr, we first obtain HDFS token for current UGI, the it can truly proxy HDFS priority. Then for each proxy user, when we create hiveClient, we obtain a Hive token for proxy user. Different user use different SparkSession.sharedState.
Then pass DFS toke to each Task, to proxy user's DFS behavior in Executors.

How was this patch tested?

manual test

AngersZhuuuu · 2019-08-07T07:13:26Z

@juliuszsompolski @wangyum Can you give some advise?

gatorsmile · 2019-08-07T16:17:45Z

ok to test

SparkQA · 2019-08-07T16:23:31Z

Test build #108773 has finished for PR 25201 at commit b414f4c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-08T11:16:47Z

Test build #108813 has finished for PR 25201 at commit f4bf959.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-08T11:24:46Z

Test build #108815 has finished for PR 25201 at commit 989b268.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-08T11:33:11Z

Test build #108816 has finished for PR 25201 at commit 1408e64.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-08T11:57:40Z

Test build #108819 has finished for PR 25201 at commit 4b4bf13.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-08T13:39:49Z

Test build #108821 has finished for PR 25201 at commit 07522c0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-08T13:50:09Z

Test build #108828 has finished for PR 25201 at commit 5135706.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-08T13:58:09Z

Test build #108830 has finished for PR 25201 at commit 76c563d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-08T15:37:34Z

Test build #108831 has finished for PR 25201 at commit dd5ce26.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-08T18:21:21Z

Test build #108839 has finished for PR 25201 at commit 997f141.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-09T01:16:11Z

Test build #108854 has finished for PR 25201 at commit 750e5be.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-09T03:08:32Z

Test build #108856 has finished for PR 25201 at commit bc7ee9e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-09T03:46:01Z

Test build #108859 has finished for PR 25201 at commit 76cd624.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-12T05:47:07Z

Test build #108948 has finished for PR 25201 at commit ac87ffc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2019-08-12T07:20:07Z

@gatorsmile
In my pr, I will create a HiveClientImple for each user to let them have their own Hive privilege.
But all old Unit Test is to start a local mode hive metastore, it conflict to my method. cause all SparkThriftServer Unit test failed

SparkQA · 2019-08-12T10:16:09Z

Test build #108966 has finished for PR 25201 at commit 3f273a6.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-12T12:17:53Z

Test build #108967 has finished for PR 25201 at commit 20eb896.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-12T16:01:12Z

Test build #108975 has finished for PR 25201 at commit 424dca3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2019-08-14T07:07:41Z

@dongjoon-hyun @gatorsmile Can you help to review and give some advise about how to add Unit test

AmplabJenkins · 2019-12-18T23:21:32Z

Can one of the admins verify this patch?

therealJacobWu · 2020-02-20T23:00:11Z

...iftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLSessionManager.scala

+          val existing = ugi.getCredentials()
+          existing.mergeAll(originalCreds)
+          ugi.addCredentials(existing)
+          sparkSqlOperationManager.sessionToTokens.put(session.getSessionHandle, tokens)


Hi @AngersZhuuuu, while applying your patch I found a bug here, session.getSessionHandle will have an exception because session is null.

Then I moved session = HiveSessionProxy.getProxy(sessionWithUGI, sessionWithUGI.getSessionUgi) to before the if statement will solve this issue.

Hi @AngersZhuuuu, while applying your patch I found a bug here, session.getSessionHandle will have an exception because session is null.

Then I moved session = HiveSessionProxy.getProxy(sessionWithUGI, sessionWithUGI.getSessionUgi) to before the if statement will solve this issue.

yea， some mistake, here is just a way to implement this. If you have interesting in this way , you can see this
spark-thriftserver/spark-thriftserver#53

github-actions · 2020-06-01T00:22:32Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

Lucusone · 2020-11-12T10:26:27Z

this PR in non kerberos environment, work fine. but in kerberos environment cannot get the hdfs token!

AngersZhuuuu · 2020-11-12T12:30:33Z

this PR in non kerberos environment, work fine. but in kerberos environment cannot get the hdfs token!

To be honest, I have never test this in non-kerberos environment since I don't have this env.
I test this in kerberos environment and it can work well. Can you show how you use it and what's the error behavior?
Fetch delegation token form HDFS won't be a blocking thing. If you are Chinese you can connect me with my gmail

Lucusone · 2020-11-13T06:48:44Z

ok

Lucusone · 2020-11-20T11:29:21Z

this pr is good, I test this pr in spark2.4.3 envrionment, this pr is base on spark3. get hdfs token is different.

sujith71955 · 2022-02-21T16:34:55Z

cc @AngersZhuuuu @gatorsmile @dongjoon-hyun @vinodkc
Can we re-open this PR since lack of impersonation is blocking the use-cases where user wants to execute queries as current users.

sujith71955 · 2022-03-19T06:13:27Z

@AngersZhuuuu We tested this patch with spark 3.x version in kerberos environment and looks good. Would be great if we can get review done from experts. Thanks

AngersZhuuuu · 2022-03-21T02:14:46Z

@AngersZhuuuu We tested this patch with spark 3.x version in kerberos environment and looks good. Would be great if we can get review done from experts. Thanks

Your env with kerberos?

weand · 2022-04-05T13:19:05Z

👍 to get the PR reopened or get the reasons for why not ?

AnhQuanTran · 2022-10-10T02:35:27Z

@AngersZhuuuu i'm facing the same this problem on spark 3.2.2. Can u tell me how to fix it. Thank you

https://stackoverflow.com/questions/73984517/spark-thrift-3-2-2-impersonate-user-facing-error-with-metastore-authen-sasl-neg

goutam-git · 2024-03-26T13:51:35Z

@AngersZhuuuu can you please provide the steps as how you have tested this patch

Angers added 2 commits July 19, 2019 20:31

[SPARK-28419] Enable STS support proxy user authendication .

118625e

for hive v2.3.5

2d5155d

dongjoon-hyun added the SQL label Jul 20, 2019

AngersZhuuuu changed the title ~~[SPARK-28419][SQL] Enable STS support proxy user's authentication .~~ [SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . Jul 24, 2019

AngersZhuuuu added 2 commits August 6, 2019 17:22

fis small spell mistake

62f5de5

for each session add a hive token

b414f4c

fix scala style

f4bf959

AngersZhuuuu added 2 commits August 8, 2019 19:17

fix scala style

bcece85

fix conflicts

989b268

fix scala style

1408e64

fix scala style

4b4bf13

fix scala style

07522c0

don't obtain hive token when null

5135706

AngersZhuuuu added 2 commits August 8, 2019 21:52

add HDFS token when security enbale

76c563d

fix scala style

5990a54

fix scala style

dd5ce26

add try catch for got hdfs token

997f141

handle Unsupport exception when getHivedelegation token

750e5be

get UnsupportedOperationException return token as null

bc7ee9e

init spark session after init hive session

76cd624

Fix scala style

ac87ffc

start UT STS without Impersonation and

3f273a6

fix scala style

20eb896

without Impersonation, support singlesession

424dca3

therealJacobWu reviewed Feb 20, 2020

View reviewed changes

github-actions bot added the Stale label Jun 1, 2020

github-actions bot closed this Jun 2, 2020

[SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . #25201

[SPARK-28419][SQL] Enable SparkThriftServer support proxy user's authentication . #25201

Conversation

AngersZhuuuu commented Jul 19, 2019 • edited

What changes were proposed in this pull request?

How was this patch tested?

AngersZhuuuu commented Aug 7, 2019

gatorsmile commented Aug 7, 2019

SparkQA commented Aug 7, 2019

SparkQA commented Aug 8, 2019

SparkQA commented Aug 8, 2019

SparkQA commented Aug 8, 2019

SparkQA commented Aug 8, 2019

SparkQA commented Aug 8, 2019

SparkQA commented Aug 8, 2019

SparkQA commented Aug 8, 2019

SparkQA commented Aug 8, 2019

SparkQA commented Aug 8, 2019

SparkQA commented Aug 9, 2019

SparkQA commented Aug 9, 2019

SparkQA commented Aug 9, 2019

SparkQA commented Aug 12, 2019

AngersZhuuuu commented Aug 12, 2019

SparkQA commented Aug 12, 2019

SparkQA commented Aug 12, 2019

SparkQA commented Aug 12, 2019

AngersZhuuuu commented Aug 14, 2019

AmplabJenkins commented Dec 18, 2019

therealJacobWu Feb 20, 2020

Choose a reason for hiding this comment

AngersZhuuuu Feb 21, 2020

Choose a reason for hiding this comment

github-actions bot commented Jun 1, 2020

Lucusone commented Nov 12, 2020

AngersZhuuuu commented Nov 12, 2020

Lucusone commented Nov 13, 2020

Lucusone commented Nov 20, 2020

sujith71955 commented Feb 21, 2022 • edited

sujith71955 commented Mar 19, 2022 • edited

AngersZhuuuu commented Mar 21, 2022

weand commented Apr 5, 2022

AnhQuanTran commented Oct 10, 2022

goutam-git commented Mar 26, 2024

AngersZhuuuu commented Jul 19, 2019 •

edited

sujith71955 commented Feb 21, 2022 •

edited

sujith71955 commented Mar 19, 2022 •

edited