[SPARK-7524][SPARK-7846]add configs for keytab and principal, pass these two configs with different way in different modes #6051

WangTaoTheTonic · 2015-05-11T11:07:41Z

As spark now supports long running service by updating tokens for namenode, but only accept parameters passed with "--k=v" format which is not very convinient. This patch add spark.* configs in properties file and system property.
--principal and --keytabl options are passed to client but when we started thrift server or spark-shell these two are also passed into the Main class (org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 and org.apache.spark.repl.Main).
In these two main class, arguments passed in will be processed with some 3rd libraries, which will lead to some error: "Invalid option: --principal" or "Unrecgnised option: --principal".
We should pass these command args in different forms, say system properties.

AmplabJenkins · 2015-05-11T11:12:10Z

Merged build triggered.

AmplabJenkins · 2015-05-11T11:12:17Z

Merged build started.

SparkQA · 2015-05-11T11:14:00Z

Test build #32382 has started for PR 6051 at commit 73afa64.

AmplabJenkins · 2015-05-11T11:17:10Z

Merged build triggered.

AmplabJenkins · 2015-05-11T11:17:17Z

Merged build started.

SparkQA · 2015-05-11T11:18:55Z

Test build #32383 has started for PR 6051 at commit 08bb4e8.

SparkQA · 2015-05-11T12:31:34Z

Test build #32383 has finished for PR 6051 at commit 08bb4e8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-11T12:31:39Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-11T12:31:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32383/
Test FAILed.

SparkQA · 2015-05-11T12:53:12Z

Test build #32382 has finished for PR 6051 at commit 73afa64.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-11T12:53:17Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-11T12:53:17Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32382/
Test PASSed.

tgravescs · 2015-05-11T14:54:52Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala

@@ -169,6 +169,8 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
    deployMode = Option(deployMode).orElse(env.get("DEPLOY_MODE")).orNull
    numExecutors = Option(numExecutors)
      .getOrElse(sparkProperties.get("spark.executor.instances").orNull)
+    keytab = Option(keytab).orElse(sparkProperties.get("spark.yarn.keytab")).orNull
+    principal = Option(principal).orElse(sparkProperties.get("spark.yarn.principal")).orNull


please document the configs since you are making them public in the running-on-yarn.md

harishreedharan · 2015-05-11T17:56:48Z

Have you actually run this one a cluster with the various options:

Use --principal + spark.yarn.keytab (and vice versa)
Both set and making sure command line gets picked up
Only command line and only spark.yarn.*

Also, I have not seen spark.internal. options elsewhere in Spark? Are there are other config options like this? I am wondering if it makes sense to just re-use the current configs? If the command line params are set, just overwrite the ones we read from the configuration?

tgravescs · 2015-05-11T18:02:34Z

No spark doesn't use the .internal. elsewhere. Please don't rename any of the ones we haven't documented as internal.

I agree that we should just expand the use of the current ones since they are going to be propagated anyway.

tgravescs · 2015-05-21T14:22:14Z

@WangTaoTheTonic can you update based on comments?

WangTaoTheTonic · 2015-05-22T09:38:01Z

@tgravescs Okay, I will do it right now.

AmplabJenkins · 2015-05-24T13:12:11Z

Build triggered.

AmplabJenkins · 2015-05-24T13:12:16Z

Build started.

SparkQA · 2015-05-24T13:14:21Z

Test build #33438 has started for PR 6051 at commit ecfe43a.

AmplabJenkins · 2015-05-24T13:22:10Z

Merged build triggered.

AmplabJenkins · 2015-05-24T13:22:17Z

Merged build started.

SparkQA · 2015-05-24T13:22:43Z

Test build #33439 has started for PR 6051 at commit ebd9ea0.

AmplabJenkins · 2015-05-24T14:33:10Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-24T14:33:11Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33439/
Test FAILed.

SparkQA · 2015-05-24T15:04:31Z

Test build #33438 has finished for PR 6051 at commit ecfe43a.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-24T15:04:36Z

Build finished. Test FAILed.

AmplabJenkins · 2015-05-24T15:04:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33438/
Test FAILed.

SparkQA · 2015-05-24T15:38:47Z

Test build #33441 has finished for PR 6051 at commit e65699a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-24T15:38:52Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-24T15:38:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33441/
Test PASSed.

WangTaoTheTonic · 2015-05-24T16:43:42Z

I updated per comments and add fix for SPARK-7846. Haven't tested as there is some problem with my cluster and will do it ASAP.

WangTaoTheTonic · 2015-05-25T04:51:19Z

@harishreedharan
I have tested per your comments, using spark-shell:

Use --principal + spark.yarn.keytab (and vice versa) (double check)
Both set and making sure command line gets picked up (check)
Only command line (check) and only spark.yarn.* (check)

It all worked fine.

WangTaoTheTonic · 2015-05-29T12:35:09Z

ping? @harishreedharan @tgravescs

tgravescs · 2015-05-29T12:40:57Z

lgtm. going to kick jenkins once more.

tgravescs · 2015-05-29T12:41:01Z

Jenkins, test this please

AmplabJenkins · 2015-05-29T12:42:11Z

Merged build triggered.

AmplabJenkins · 2015-05-29T12:42:19Z

Merged build started.

SparkQA · 2015-05-29T12:44:26Z

Test build #33736 has started for PR 6051 at commit e65699a.

SparkQA · 2015-05-29T14:33:54Z

Test build #33736 has finished for PR 6051 at commit e65699a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-29T14:33:59Z

Merged build finished. Test PASSed.

tgravescs · 2015-05-29T14:47:48Z

@WangTaoTheTonic can you please fix the description on this pr. Remove the [YARN]1 from heading.

WangTaoTheTonic · 2015-05-29T16:01:12Z

@tgravescs Done.

harishreedharan · 2015-05-29T23:18:19Z

Sorry, I am late to this to this one. Could you create a followup PR to update the security.md file with info about running in client vs cluster modes and that this is supported only in cluster mode via command line params.

@tgravescs - If 1.4 gets released without this patch, there will be a regression in client mode (--principal and --keytab won't work in client mode). Should this + docs for this one also go into 1.4?

tgravescs · 2015-05-30T00:36:05Z

@harishreedharan I'm pretty sure the --keytab and --principal options still work in client mode to SparkSubmit. Its simply the way it passes it on to the client. in yarn client mode it uses the config and in yarn cluster mode it uses the command line options.

OptionAssigner(args.principal, YARN, CLIENT, sysProp = "spark.yarn.principal"),
OptionAssigner(args.keytab, YARN, CLIENT, sysProp = "spark.yarn.keytab"),

I'm building now and will test it to double check. Did you try it yourself?

That would not be a blocker for 1.4 though. We would revert and fix.

harishreedharan · 2015-05-30T00:42:35Z

Ah, right - my bad! I wasn't reading the code properly. I will try it out later today to confirm.

WangTaoTheTonic · 2015-05-30T08:24:12Z

@harishreedharan How about the confirm? Should we do some changes in branh 1.4?

…these two configs with different way in different modes * As spark now supports long running service by updating tokens for namenode, but only accept parameters passed with "--k=v" format which is not very convinient. This patch add spark.* configs in properties file and system property. * --principal and --keytabl options are passed to client but when we started thrift server or spark-shell these two are also passed into the Main class (org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 and org.apache.spark.repl.Main). In these two main class, arguments passed in will be processed with some 3rd libraries, which will lead to some error: "Invalid option: --principal" or "Unrecgnised option: --principal". We should pass these command args in different forms, say system properties. Author: WangTaoTheTonic <wangtao111@huawei.com> Closes apache#6051 from WangTaoTheTonic/SPARK-7524 and squashes the following commits: e65699a [WangTaoTheTonic] change logic to loadEnvironments ebd9ea0 [WangTaoTheTonic] merge master ecfe43a [WangTaoTheTonic] pass keytab and principal seperately in different mode 33a7f40 [WangTaoTheTonic] expand the use of the current configs 08bb4e8 [WangTaoTheTonic] fix wrong cite 73afa64 [WangTaoTheTonic] add configs for keytab and principal, move originals to internal

…these two configs with different way in different modes * As spark now supports long running service by updating tokens for namenode, but only accept parameters passed with "--k=v" format which is not very convinient. This patch add spark.* configs in properties file and system property. * --principal and --keytabl options are passed to client but when we started thrift server or spark-shell these two are also passed into the Main class (org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 and org.apache.spark.repl.Main). In these two main class, arguments passed in will be processed with some 3rd libraries, which will lead to some error: "Invalid option: --principal" or "Unrecgnised option: --principal". We should pass these command args in different forms, say system properties. Author: WangTaoTheTonic <wangtao111@huawei.com> Closes apache#6051 from WangTaoTheTonic/SPARK-7524 and squashes the following commits: e65699a [WangTaoTheTonic] change logic to loadEnvironments ebd9ea0 [WangTaoTheTonic] merge master ecfe43a [WangTaoTheTonic] pass keytab and principal seperately in different mode 33a7f40 [WangTaoTheTonic] expand the use of the current configs 08bb4e8 [WangTaoTheTonic] fix wrong cite 73afa64 [WangTaoTheTonic] add configs for keytab and principal, move originals to internal Conflicts: docs/running-on-yarn.md

add configs for keytab and principal, move originals to internal

73afa64

fix wrong cite

08bb4e8

tgravescs reviewed May 11, 2015
View reviewed changes

WangTaoTheTonic added 2 commits May 24, 2015 21:01

expand the use of the current configs

33a7f40

pass keytab and principal seperately in different mode

ecfe43a

WangTaoTheTonic changed the title ~~[SPARK-7524][YARN]add configs for keytab and principal, move originals to internal~~ [SPARK-7524][SPARK-7846][YARN]add configs for keytab and principal, move originals to internal May 24, 2015

merge master

ebd9ea0

WangTaoTheTonic changed the title ~~[SPARK-7524][SPARK-7846]1. add configs for keytab and principal 2.pass these two configs with different way in different modes~~ [SPARK-7524][SPARK-7846]add configs for keytab and principal, pass these two configs with different way in different modes May 29, 2015

asfgit closed this in a51b133 May 29, 2015

[SPARK-7524][SPARK-7846]add configs for keytab and principal, pass these two configs with different way in different modes #6051

[SPARK-7524][SPARK-7846]add configs for keytab and principal, pass these two configs with different way in different modes #6051

Conversation

WangTaoTheTonic commented May 11, 2015

AmplabJenkins commented May 11, 2015

AmplabJenkins commented May 11, 2015

SparkQA commented May 11, 2015

AmplabJenkins commented May 11, 2015

AmplabJenkins commented May 11, 2015

SparkQA commented May 11, 2015

SparkQA commented May 11, 2015

AmplabJenkins commented May 11, 2015

AmplabJenkins commented May 11, 2015

SparkQA commented May 11, 2015

AmplabJenkins commented May 11, 2015

AmplabJenkins commented May 11, 2015

tgravescs May 11, 2015

Choose a reason for hiding this comment

harishreedharan commented May 11, 2015

tgravescs commented May 11, 2015

tgravescs commented May 21, 2015

WangTaoTheTonic commented May 22, 2015

AmplabJenkins commented May 24, 2015

AmplabJenkins commented May 24, 2015

SparkQA commented May 24, 2015

AmplabJenkins commented May 24, 2015

AmplabJenkins commented May 24, 2015

SparkQA commented May 24, 2015

AmplabJenkins commented May 24, 2015

AmplabJenkins commented May 24, 2015

SparkQA commented May 24, 2015

AmplabJenkins commented May 24, 2015

AmplabJenkins commented May 24, 2015

SparkQA commented May 24, 2015

AmplabJenkins commented May 24, 2015

AmplabJenkins commented May 24, 2015

WangTaoTheTonic commented May 24, 2015

WangTaoTheTonic commented May 25, 2015

WangTaoTheTonic commented May 29, 2015

tgravescs commented May 29, 2015

tgravescs commented May 29, 2015

AmplabJenkins commented May 29, 2015

AmplabJenkins commented May 29, 2015

SparkQA commented May 29, 2015

SparkQA commented May 29, 2015

AmplabJenkins commented May 29, 2015

tgravescs commented May 29, 2015

WangTaoTheTonic commented May 29, 2015

harishreedharan commented May 29, 2015

tgravescs commented May 30, 2015

harishreedharan commented May 30, 2015

WangTaoTheTonic commented May 30, 2015