[SPARK-10181][SQL] Do kerberos login for credentials during hive client initialization #9272

yolandagao · 2015-10-25T20:24:40Z

On driver process start up, UserGroupInformation.loginUserFromKeytab is called with the principal and keytab passed in, and therefore static var UserGroupInfomation,loginUser is set to that principal with kerberos credentials saved in its private credential set, and all threads within the driver process are supposed to see and use this login credentials to authenticate with Hive and Hadoop. However, because of IsolatedClientLoader, UserGroupInformation class is not shared for hive metastore clients, and instead it is loaded separately and of course not able to see the prepared kerberos login credentials in the main thread.

The first proposed fix would cause other classloader conflict errors, and is not an appropriate solution. This new change does kerberos login during hive client initialization, which will make credentials ready for the particular hive client instance.

@yhuai Please take a look and let me know. If you are not the right person to talk to, could you point me to someone responsible for this?

…redentials

… login credentials" This reverts commit 59253b3.

…zation

yhuai · 2015-10-26T19:28:58Z

This this please.

yhuai · 2015-10-26T19:30:45Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

@@ -520,6 +520,8 @@ object SparkSubmit {
      }
      if (args.principal != null) {
        require(args.keytab != null, "Keytab must be specified when the keytab is specified")
+        sysProps.put("spark.yarn.keytab", args.keytab)
+        sysProps.put("spark.yarn.principal", args.principal)


@harishreedharan I see you changed this part of code last time. If we want to pass these two arguments to Spark SQL, what is the recommended way?

@harishreedharan Hi Hari, could you let us know the preferred way to pass principal and keytab parameters from spark submit to spark sql? waiting for your response to proceed. Thank you!

We might want to look at yarn Client.scala setupCredentials() since its doing something pretty similar it looks like.

yhuai · 2015-11-02T02:25:57Z

@yolandagao Is 40d3c67 for the same issue (for a similar issue)? @steveloughran Can you take a look at this one? I am not sure if it is the same issue that you addressed in SPARK-11265.

yolandagao · 2015-11-02T03:47:23Z

@yhuai Hi Yin, SPARK-11265 is a different issue - in yarn Client.scala code, hive metastore token is obtained when kerberos is enabled to set up AM container launch context, during which Hive and HiveConf instantiation failed due to a new change in Hive 1.2.1.
YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf, credentials)
obtainTokenForHiveMetastore(hadoopConf, credentials)
obtainTokenForHBase(hadoopConf, credentials)
....
setupSecurityToken(amContainer)
UserGroupInformation.getCurrentUser().addCredentials(credentials)

SPARK-11265 fix will not solve the problem here. In yarn-client mode, Hive client with IsolatedClientLoader has no access to these delegation tokens and kerberos tickets (set up via UGI.loginUserFromKeytab in SparkSubmit), as it has a pure new UserGroupInformation class loaded.

holdenk · 2015-11-03T01:00:51Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

@@ -520,6 +520,8 @@ object SparkSubmit {
      }
      if (args.principal != null) {
        require(args.keytab != null, "Keytab must be specified when the keytab is specified")


I know this line wasn't changed in this PR but since we are in this file anyways could we maybe cleanup this error message (I think it is meant to say "Keytab must be specified when the kerberos principal is specified"

Sure. Corrected the error message. Thanks!

…ialization

yolandagao · 2015-11-03T08:36:36Z

Hi folks,
Yarn Client.scala checks the setting from argStrings (passed from SparkSubmit) and sparkConf (which loads java system properties starting with spark.* as well). The args settings (--principal, --keytab) will not be available in spark sql, but we can take advantage of SparkConf with system properties. Updated the pull request and tested. Please advise if this is what you are looking for, Thank you!

steveloughran · 2015-11-04T11:08:44Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala

+
+    val sparkConf = new SparkConf
+    if (sparkConf.contains("spark.yarn.principal") && sparkConf.contains("spark.yarn.keytab")) {
+      UserGroupInformation.loginUserFromKeytab(


before calling this, actually verify that the keytab file exists and fail with a message including the property name, to help people debug the problem. UGI internal exceptions are rarely informative enough

steveloughran · 2015-11-04T11:12:08Z

This is unrelated to the SPARK-11265 patch; that's all about getting reflection to find the right methods. This is about UGI setup.

steveloughran · 2015-11-04T11:12:48Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala

+    val sparkConf = new SparkConf
+    if (sparkConf.contains("spark.yarn.principal") && sparkConf.contains("spark.yarn.keytab")) {
+      UserGroupInformation.loginUserFromKeytab(
+        sparkConf.get("spark.yarn.principal"),


actually, you should call SparkHadoopUtil.get.loginUserFromKeytab(principalName, keytabFilename; maybe the check could be added there.

@steveloughran Good point. Better to check the existence of the keytab file before make the login call, as if the keytab doesn't exist the UGI call will definitely fail but with some indirect message like "login failed... no keys found..." ect. Added the check.

However, calling SparkHadoopUtil.get.loginUserFromKeytab instead of UserGroupInformation.loginUserFromKeytab in ClientWrapper will not solve the problem as SparkHadoopUtil is shared and the UserGroupInformation class it includes is not the same one used by SessionState.start in ClientWrapper. Therefore, the program still fails with no tgt exception when connecting to metastore. Also not able to replace the UGI call in SparkSubmit either, as incorrect type of SparkHadoopUtil instance might get created due to yarn mode isn't set in the system until it flows to yarn Client.scala.

OK. just include the check (actually, UGI itself should do that check shouldn't it? Lazy)

…ialization V3

…ialization V4

yolandagao · 2015-11-05T10:07:51Z

Updated and tested the change - in yarn client and cluster mode, with and without keytab/principal parameters. Please take a look. Thank you!

marmbrus · 2015-11-07T01:15:43Z

ok to test

SparkQA · 2015-11-08T21:44:35Z

Test build #45308 has finished for PR 9272 at commit 7d09f5d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

yolandagao · 2015-11-09T05:45:09Z

The org.apache.spark.sql.hive.thriftserver.CliSuite timeout failure seems not related to the change but some environment issue. Ran the test suite on my mac machine several times with the same sbt command, and it all passed.

Can we retest?

holdenk · 2015-11-09T18:46:02Z

@yolandagao you can try saying "jenkins retest this please" in a comment (I know if you've been whitelisted it will trigger a retest less certain if not yet whitelisted).

SparkQA · 2015-11-09T22:28:59Z

Test build #45392 has finished for PR 9272 at commit 7d09f5d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yhuai · 2015-11-09T22:43:06Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala

@@ -150,6 +152,21 @@ private[hive] class ClientWrapper(
    val original = Thread.currentThread().getContextClassLoader
    // Switch to the initClassLoader.
    Thread.currentThread().setContextClassLoader(initClassLoader)
+
+    val sparkConf = new SparkConf


Instead of creating a new Spark Conf, can we use SparkEnv.get.conf to get the spark conf associated with the current spark context?

@yhuai Sorry for the late response. I did the testing after changing to SparkEnv.get.conf but it didn't work. The reason is that Yarn Client.scala resets property spark.yarn.keytab by appending some random strings to keytab file name during setupCredentials, which will be used as the link name in distributed cache. I think the one for the link name should be actually separated from the original keytab setting, e.g. using different property names.

@yolandagao Thanks for the explanation. Can you add comments to your code (including why we need to put those confs to sysProps and why we need to create a new SparkConf at here)? Basically, we need to document the flow of how these confs get propagated. Otherwise, it is not obvious why we need to do this change. Thanks!

@yhuai Sure. I should have done this earlier, to make everything clearer:) Added some comments there, and please help review. Thank you!

…ation

yhuai · 2015-11-14T21:07:30Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala

+    // SparkConf is needed for the original value of spark.yarn.keytab specified by user,
+    // as yarn.Client resets it for the link name in distributed cache
+    val sparkConf = new SparkConf
+    if (sparkConf.contains("spark.yarn.principal") && sparkConf.contains("spark.yarn.keytab")) {


Let's make it clear that we set these two settings in SparkSubmit.

yhuai · 2015-11-14T21:08:54Z

@yolandagao Thank you for the update. Overall looks good. Left two comments.

yhuai · 2015-11-14T21:14:19Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala

@@ -20,6 +20,8 @@ package org.apache.spark.sql.hive.client
 import java.io.{File, PrintStream}
 import java.util.{Map => JMap}

+import org.apache.hadoop.security.UserGroupInformation


Let's move this import down to the place where we have other hadoop related imports. https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide#SparkCodeStyleGuide-Imports is the doc about import ordering.

SparkQA · 2015-11-14T23:30:39Z

Test build #45937 has finished for PR 9272 at commit 1fbc372.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yolandagao · 2015-11-15T03:06:22Z

Thank you Yin for the review. Updated the comments accordingly.

yhuai · 2015-11-15T03:11:14Z

LGTM pending jenkins.

SparkQA · 2015-11-15T05:57:16Z

Test build #45944 has finished for PR 9272 at commit caf51a7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yhuai · 2015-11-15T22:53:31Z

Thanks! Merging to master and branch 1.6.

…nt initialization On driver process start up, UserGroupInformation.loginUserFromKeytab is called with the principal and keytab passed in, and therefore static var UserGroupInfomation,loginUser is set to that principal with kerberos credentials saved in its private credential set, and all threads within the driver process are supposed to see and use this login credentials to authenticate with Hive and Hadoop. However, because of IsolatedClientLoader, UserGroupInformation class is not shared for hive metastore clients, and instead it is loaded separately and of course not able to see the prepared kerberos login credentials in the main thread. The first proposed fix would cause other classloader conflict errors, and is not an appropriate solution. This new change does kerberos login during hive client initialization, which will make credentials ready for the particular hive client instance. yhuai Please take a look and let me know. If you are not the right person to talk to, could you point me to someone responsible for this? Author: Yu Gao <ygao@us.ibm.com> Author: gaoyu <gaoyu@gaoyu-macbookpro.roam.corp.google.com> Author: Yu Gao <crystalgaoyu@gmail.com> Closes #9272 from yolandagao/master. (cherry picked from commit 72c1d68) Signed-off-by: Yin Huai <yhuai@databricks.com>

yhuai · 2015-11-16T18:30:14Z

Also merged to branch-1.5.

yolandagao added 6 commits October 20, 2015 18:23

SPARK-10181 UserGroupInformation class needs to be shared for login c…

59253b3

…redentials

Merge remote branch 'upstream/master'

3e79e94

Revert "SPARK-10181 UserGroupInformation class needs to be shared for…

9efa819

… login credentials" This reverts commit 59253b3.

Merge remote branch 'upstream/master'

7934585

SPARK-10181 Do kerberos login for credentials in hive client initiali…

30a7d05

…zation

Merge remote branch 'upstream/master'

836bb8b

yhuai reviewed Oct 26, 2015
View reviewed changes

holdenk reviewed Nov 3, 2015
View reviewed changes

yolandagao added 2 commits November 2, 2015 20:16

Merge remote branch 'upstream/master'

7ec246f

SPARK-10181 Do kerberos login for credentials during hive client init…

1fe7593

…ialization

steveloughran reviewed Nov 4, 2015
View reviewed changes

yolandagao added 5 commits November 4, 2015 14:33

Merge remote branch 'upstream/master'

3f0fee9

Merge remote branch 'upstream/master'

06ee550

SPARK-10181 Do kerberos login for credentials during hive client init…

906b3cb

…ialization V3

Merge remote branch 'upstream/master'

ef927e7

SPARK-10181 Do kerberos login for credentials during hive client init…

7d09f5d

…ialization V4

yhuai reviewed Nov 9, 2015
View reviewed changes

Merge remote-tracking branch 'upstream/master'

36d5ef8

gaoyu and others added 2 commits November 14, 2015 11:54

Merge remote-tracking branch 'upstream/master'

cdb3aa5

SPARK-10181 Add comments explaining the keytab principal confs propag…

1fbc372

…ation

yhuai reviewed Nov 14, 2015
View reviewed changes

SPARK-10181 Add more comments for clarification

caf51a7

asfgit closed this in 72c1d68 Nov 15, 2015

jerryshao mentioned this pull request Feb 17, 2017

[SPARK-19038][YARN] Avoid overwriting keytab configuration in yarn-client #16923

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-10181][SQL] Do kerberos login for credentials during hive client initialization #9272

[SPARK-10181][SQL] Do kerberos login for credentials during hive client initialization #9272

yolandagao commented Oct 25, 2015

yhuai commented Oct 26, 2015

yhuai Oct 26, 2015

yolandagao Nov 2, 2015

holdenk Nov 3, 2015

yhuai commented Nov 2, 2015

yolandagao commented Nov 2, 2015

holdenk Nov 3, 2015

yolandagao Nov 3, 2015

yolandagao commented Nov 3, 2015

steveloughran Nov 4, 2015

steveloughran commented Nov 4, 2015

steveloughran Nov 4, 2015

yolandagao Nov 5, 2015

steveloughran Nov 5, 2015

yolandagao commented Nov 5, 2015

marmbrus commented Nov 7, 2015

SparkQA commented Nov 8, 2015

yolandagao commented Nov 9, 2015

holdenk commented Nov 9, 2015

SparkQA commented Nov 9, 2015

yhuai Nov 9, 2015

yolandagao Nov 13, 2015

yhuai Nov 13, 2015

yolandagao Nov 14, 2015

yhuai Nov 14, 2015

yhuai commented Nov 14, 2015

yhuai Nov 14, 2015

SparkQA commented Nov 14, 2015

yolandagao commented Nov 15, 2015

yhuai commented Nov 15, 2015

SparkQA commented Nov 15, 2015

yhuai commented Nov 15, 2015

yhuai commented Nov 16, 2015

[SPARK-10181][SQL] Do kerberos login for credentials during hive client initialization #9272

[SPARK-10181][SQL] Do kerberos login for credentials during hive client initialization #9272

Conversation

yolandagao commented Oct 25, 2015

yhuai commented Oct 26, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yhuai commented Nov 2, 2015

yolandagao commented Nov 2, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yolandagao commented Nov 3, 2015

Choose a reason for hiding this comment

steveloughran commented Nov 4, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yolandagao commented Nov 5, 2015

marmbrus commented Nov 7, 2015

SparkQA commented Nov 8, 2015

yolandagao commented Nov 9, 2015

holdenk commented Nov 9, 2015

SparkQA commented Nov 9, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yhuai commented Nov 14, 2015

Choose a reason for hiding this comment

SparkQA commented Nov 14, 2015

yolandagao commented Nov 15, 2015

yhuai commented Nov 15, 2015

SparkQA commented Nov 15, 2015

yhuai commented Nov 15, 2015

yhuai commented Nov 16, 2015