New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-23476][CORE] Generate secret in local mode when authentication on #20652

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
5 participants
@gaborgsomogyi
Contributor

gaborgsomogyi commented Feb 21, 2018

What changes were proposed in this pull request?

If spark is run with "spark.authenticate=true", then it will fail to start in local mode.

This PR generates secret in local mode when authentication on.

How was this patch tested?

Modified existing unit test.
Manually started spark-shell.

@gaborgsomogyi

This comment has been minimized.

Contributor

gaborgsomogyi commented Feb 21, 2018

cc @vanzin

@vanzin

This comment has been minimized.

Contributor

vanzin commented Feb 21, 2018

ok to test

@@ -529,7 +529,8 @@ private[spark] class SecurityManager(
return
}
if (sparkConf.get(SparkLauncher.SPARK_MASTER, null) != "yarn") {
val master = sparkConf.get(SparkLauncher.SPARK_MASTER, "")

This comment has been minimized.

@vanzin

vanzin Feb 21, 2018

Contributor

I'm not sure this will work for local-cluster. Try local-cluster[2, 1, 1024] as the master and see what happens.

This comment has been minimized.

@gaborgsomogyi

gaborgsomogyi Feb 21, 2018

Contributor

Nice catch! I've enhanced the test and fixed the issue.

@SparkQA

This comment has been minimized.

SparkQA commented Feb 21, 2018

Test build #87593 has finished for PR 20652 at commit cf35ff3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@vanzin

This comment has been minimized.

Contributor

vanzin commented Feb 22, 2018

LGTM, but the title should say [core] (not spark-shell).

@gaborgsomogyi gaborgsomogyi changed the title from [SPARK-23476][SPARK-SHELL] Generate secret in local mode when authentication on to [SPARK-23476][CORE] Generate secret in local mode when authentication on Feb 22, 2018

@SparkQA

This comment has been minimized.

SparkQA commented Feb 22, 2018

Test build #87601 has finished for PR 20652 at commit 3f49371.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@vanzin

This comment has been minimized.

Contributor

vanzin commented Feb 22, 2018

retest this please

@SparkQA

This comment has been minimized.

SparkQA commented Feb 22, 2018

Test build #87604 has finished for PR 20652 at commit 3f49371.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@squito

This comment has been minimized.

Contributor

squito commented Feb 22, 2018

lgtm

@jiangxb1987

LGTM

("yarn", true),
("local", true),
("local[*]", true),
("local[1,2]", true),

This comment has been minimized.

@jiangxb1987

jiangxb1987 Feb 22, 2018

Contributor

nit: local[1,2] -> local[1, 2]

This comment has been minimized.

@gaborgsomogyi

gaborgsomogyi Feb 22, 2018

Contributor

Fixed.

@SparkQA

This comment has been minimized.

SparkQA commented Feb 22, 2018

Test build #87619 has finished for PR 20652 at commit e01feef.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@vanzin

This comment has been minimized.

Contributor

vanzin commented Feb 22, 2018

Merging to master.

@asfgit asfgit closed this in c5abb3c Feb 22, 2018

wangyum added a commit to wangyum/spark that referenced this pull request Mar 5, 2018

[SPARK-23476][CORE] Generate secret in local mode when authentication on
## What changes were proposed in this pull request?

If spark is run with "spark.authenticate=true", then it will fail to start in local mode.

This PR generates secret in local mode when authentication on.

## How was this patch tested?

Modified existing unit test.
Manually started spark-shell.

Author: Gabor Somogyi <gabor.g.somogyi@gmail.com>

Closes apache#20652 from gaborgsomogyi/SPARK-23476.

jcuquemelle pushed a commit to jcuquemelle/spark that referenced this pull request Mar 11, 2018

[SPARK-23476][CORE] Generate secret in local mode when authentication on
## What changes were proposed in this pull request?

If spark is run with "spark.authenticate=true", then it will fail to start in local mode.

This PR generates secret in local mode when authentication on.

## How was this patch tested?

Modified existing unit test.
Manually started spark-shell.

Author: Gabor Somogyi <gabor.g.somogyi@gmail.com>

Closes apache#20652 from gaborgsomogyi/SPARK-23476.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment