[SPARK-26766][CORE] Remove the list of filesystems from HadoopDelegationTokenProvider.obtainDelegationTokens #23698
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ionTokenProvider.obtainDelegationTokens
|
Test build #101891 has finished for PR 23698 at commit
|
|
Test build #101895 has finished for PR 23698 at commit
|
vanzin
reviewed
Jan 30, 2019
core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala
Outdated
Show resolved
Hide resolved
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala
Show resolved
Hide resolved
* Config parameter deprecation
* Return defaultFS all the time
* get("spark.master", null)
|
Test build #101959 has finished for PR 23698 at commit
|
|
Test build #101960 has finished for PR 23698 at commit
|
vanzin
reviewed
Jan 31, 2019
core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/internal/config/package.scala
Outdated
Show resolved
Hide resolved
|
Still missing the doc update. |
* Doc update * Param deprecation
Yeah, left from the last commit :/ Now updated. |
|
Test build #101994 has finished for PR 23698 at commit
|
vanzin
reviewed
Feb 6, 2019
* Simplified hadoopFSsToAccess * Moved doc to generic area
|
Test build #102077 has finished for PR 23698 at commit
|
|
retest this please |
|
Test build #102082 has finished for PR 23698 at commit
|
|
Merging to master. |
jackylee-ch
pushed a commit
to jackylee-ch/spark
that referenced
this pull request
Feb 18, 2019
…ionTokenProvider.obtainDelegationTokens ## What changes were proposed in this pull request? Delegation token providers interface now has a parameter `fileSystems` but this is needed only for `HadoopFSDelegationTokenProvider`. In this PR I've addressed this issue in the following way: * Removed `fileSystems` parameter from `HadoopDelegationTokenProvider` * Moved `YarnSparkHadoopUtil.hadoopFSsToAccess` into `HadoopFSDelegationTokenProvider` * Moved `spark.yarn.stagingDir` into core * Moved `spark.yarn.access.namenodes` into core and renamed to `spark.kerberos.access.namenodes` * Moved `spark.yarn.access.hadoopFileSystems` into core and renamed to `spark.kerberos.access.hadoopFileSystems` ## How was this patch tested? Existing unit tests. Closes apache#23698 from gaborgsomogyi/SPARK-26766. Authored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Delegation token providers interface now has a parameter
fileSystemsbut this is needed only forHadoopFSDelegationTokenProvider.In this PR I've addressed this issue in the following way:
fileSystemsparameter fromHadoopDelegationTokenProviderYarnSparkHadoopUtil.hadoopFSsToAccessintoHadoopFSDelegationTokenProviderspark.yarn.stagingDirinto corespark.yarn.access.namenodesinto core and renamed tospark.kerberos.access.namenodesspark.yarn.access.hadoopFileSystemsinto core and renamed tospark.kerberos.access.hadoopFileSystemsHow was this patch tested?
Existing unit tests.