[SPARK-8688][YARN]Bug fix: disable the cache fs to gain the HDFS connection. #7069
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Test build #35935 has started for PR 7069 at commit |
| */ | ||
| private[spark] def getDiscachedConf(hadoopConf: Configuration, path: Path): Configuration = { | ||
| val newConf = new Configuration(hadoopConf) | ||
| val confKey = s"fs.${path.toUri.getScheme}.impl.disable.cache" |
There was a problem hiding this comment.
"discached" isn't a word so I'd suggest ...ConfBypassingFSCache? wordy, but accurate
There was a problem hiding this comment.
Ok, rename it to be getConfBypassingFSCache
|
Merged build triggered. |
|
Test build #35935 has finished for PR 7069 at commit
|
|
Merged build finished. Test PASSed. |
|
Merged build started. |
|
Test build #35937 has started for PR 7069 at commit |
|
Test build #35937 has finished for PR 7069 at commit
|
|
Merged build finished. Test PASSed. |
| @@ -65,6 +65,8 @@ private[yarn] class AMDelegationTokenRenewer( | |||
| sparkConf.getInt("spark.yarn.credentials.file.retention.days", 5) | |||
| private val numFilesToKeep = | |||
| sparkConf.getInt("spark.yarn.credentials.file.retention.count", 5) | |||
| private val discachedConfiguration = | |||
There was a problem hiding this comment.
echoing @srowen's comment, can you rename this? I don't understand what discached means
There was a problem hiding this comment.
I would just call this freshHadoopConf to be more explicit
|
Merged build triggered. |
|
Merged build started. |
|
Test build #36074 has started for PR 7069 at commit |
|
Test build #36074 has finished for PR 7069 at commit
|
|
Merged build finished. Test PASSed. |
| * This is to prevent the DFSClient from using an old cached token to connect to the NameNode. | ||
| */ | ||
| private[spark] | ||
| def getConfBypassingFSCache(hadoopConf: Configuration, path: Path): Configuration = { |
There was a problem hiding this comment.
One line, and this does not need a Path arg but just a scheme, perhaps.
There was a problem hiding this comment.
if it doesn't fit on one line:
private[spark] def getConfBypassingFSCache(
hadoopConf: Configuration,
scheme: String): Configuration = {
...
}
|
Looks great. I think this is basically ready for merge. @harishreedharan can you do a sign off? |
|
Merged build triggered. |
|
Merged build started. |
|
Test build #36221 has started for PR 7069 at commit |
|
Test build #36221 has finished for PR 7069 at commit
|
|
Merged build finished. Test PASSed. |
|
I am on vacation till Sunday. I can take a look after. If you want to merge On Tuesday, June 30, 2015, UCB AMPLab notifications@github.com wrote:
Thanks, |
|
Alright then, I'm merging this into master. Thanks @SaintBacchus. |
If
fs.hdfs.impl.disable.cachewasfalse(default),FileSystemwill use the cachedDFSClientwhich use old token.AMDelegationTokenRenewer
Although the
credentialshad the new Token, but it still use the cached client and old token.So It's better to set the
fs.hdfs.impl.disable.cacheastrueto avoid token expired.Jira