Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-22319][Core] call loginUserFromKeytab before accessing hdfs #19540

Closed
wants to merge 1 commit into from
Closed

[SPARK-22319][Core] call loginUserFromKeytab before accessing hdfs #19540

wants to merge 1 commit into from

Conversation

sjrand
Copy link
Contributor

@sjrand sjrand commented Oct 20, 2017

What changes were proposed in this pull request?

In SparkSubmit, call loginUserFromKeytab before attempting to make RPC calls to the NameNode.

How was this patch tested?

I manually tested this patch by:

  1. Confirming that my Spark application failed to launch with the error reported in https://issues.apache.org/jira/browse/SPARK-22319.
  2. Applying this patch and confirming that the app no longer fails to launch, even when I have not manually run kinit on the host.

Presumably we also want integration tests for secure clusters so that we catch this sort of thing. I'm happy to take a shot at this if it's feasible and someone can point me in the right direction.

@jerryshao
Copy link
Contributor

Thanks for the fix! I didn't test on secure cluster when did glob path support, so I didn't realize such issue.

@jerryshao
Copy link
Contributor

ok to test.

@jerryshao
Copy link
Contributor

I think branch 2.2 also has similar issue when fetching resources from remote secure HDFS.

@SparkQA
Copy link

SparkQA commented Oct 20, 2017

Test build #82924 has finished for PR 19540 at commit 08240f3.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile
Copy link
Member

retest this please

@SparkQA
Copy link

SparkQA commented Oct 20, 2017

Test build #82927 has finished for PR 19540 at commit 08240f3.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987
Copy link
Contributor

Could you check why the test cases failed? Thanks!

@sjrand
Copy link
Contributor Author

sjrand commented Oct 22, 2017

I ran DistributedSuite locally via testOnly org.apache.spark.DistributedSuite from the sbt console and all tests passed. Is it possible that the failures are flakes?

@jiangxb1987
Copy link
Contributor

retest this please

@SparkQA
Copy link

SparkQA commented Oct 22, 2017

Test build #82954 has finished for PR 19540 at commit 08240f3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

jiangxb1987
jiangxb1987 approved these changes Oct 22, 2017
View reviewed changes
Copy link
Contributor

@jiangxb1987 jiangxb1987 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, @jerryshao WDYT?

@jerryshao
Copy link
Contributor

LGTM, merging to master and branch 2.2.

@asfgit asfgit closed this in 57accf6 Oct 23, 2017
@jerryshao
Copy link
Contributor

@sjrand , can you please create another PR against branch-2.2, it is not auto-mergeable, thanks!

@sjrand sjrand mentioned this pull request Oct 23, 2017
Closed
@sjrand
Copy link
Contributor Author

sjrand commented Oct 23, 2017

Thanks @jerryshao and @jiangxb1987. Created #19554 for branch-2.2.

asfgit pushed a commit that referenced this pull request Oct 23, 2017
@sjrand @jerryshao
[SPARK-22319][CORE][BACKPORT-2.2] call loginUserFromKeytab before acc…
bf8163f 
…essing hdfs

In SparkSubmit, call loginUserFromKeytab before attempting to make RPC calls to the NameNode.

Same as ##19540, but for branch-2.2.

Manually tested for master as described in #19540.

Author: Steven Rand <srand@palantir.com>

Closes #19554 from sjrand/SPARK-22319-branch-2.2.

Change-Id: Ic550a818fd6a3f38b356ac48029942d463738458
@GulajavaMinistudio GulajavaMinistudio mentioned this pull request Oct 23, 2017
Merged
MatthewRBruce pushed a commit to Shopify/spark that referenced this pull request Jul 31, 2018
@sjrand @MatthewRBruce
[SPARK-22319][CORE][BACKPORT-2.2] call loginUserFromKeytab before acc…
9f724cd 
…essing hdfs

In SparkSubmit, call loginUserFromKeytab before attempting to make RPC calls to the NameNode.

Same as #apache#19540, but for branch-2.2.

Manually tested for master as described in apache#19540.

Author: Steven Rand <srand@palantir.com>

Closes apache#19554 from sjrand/SPARK-22319-branch-2.2.

Change-Id: Ic550a818fd6a3f38b356ac48029942d463738458
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jiangxb1987 jiangxb1987 jiangxb1987 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@sjrand @jerryshao @SparkQA @gatorsmile @jiangxb1987