-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-21376][YARN] Fix yarn client token expire issue when cleaning the staging files in long running scenario #18617
Conversation
…g token expire issue Change-Id: I851f829d38ab93a4caafb7ca029d34da76f92f76
Test build #79577 has finished for PR 18617 at commit
|
@tgravescs @vanzin can you please help to review, thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really wish YARN-2261 was a thing that existed...
def cleanupStagingDirInternal(): Unit = { | ||
val stagingDirPath = new Path(appStagingBaseDir, getAppStagingDir(appId)) | ||
try { | ||
val preserveFiles = sparkConf.get(PRESERVE_STAGING_FILES) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could do this check before even trying anything else in this method (just return early).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, I will change the code.
Change-Id: I1b61dd676a5344c8e69a3d7d5268aa343bd1bf75
Test build #79593 has finished for PR 18617 at commit
|
Merging to master. |
Hi, All. |
…the staging files in long running scenario ## What changes were proposed in this pull request? This issue happens in long running application with yarn cluster mode, because yarn#client doesn't sync token with AM, so it will always keep the initial token, this token may be expired in the long running scenario, so when yarn#client tries to clean up staging directory after application finished, it will use this expired token and meet token expire issue. ## How was this patch tested? Manual verification is secure cluster. Author: jerryshao <sshao@hortonworks.com> Closes #18617 from jerryshao/SPARK-21376. (cherry picked from commit cb8d5cc)
…the staging files in long running scenario ## What changes were proposed in this pull request? This issue happens in long running application with yarn cluster mode, because yarn#client doesn't sync token with AM, so it will always keep the initial token, this token may be expired in the long running scenario, so when yarn#client tries to clean up staging directory after application finished, it will use this expired token and meet token expire issue. ## How was this patch tested? Manual verification is secure cluster. Author: jerryshao <sshao@hortonworks.com> Closes apache#18617 from jerryshao/SPARK-21376. (cherry picked from commit cb8d5cc)
What changes were proposed in this pull request?
This issue happens in long running application with yarn cluster mode, because yarn#client doesn't sync token with AM, so it will always keep the initial token, this token may be expired in the long running scenario, so when yarn#client tries to clean up staging directory after application finished, it will use this expired token and meet token expire issue.
How was this patch tested?
Manual verification is secure cluster.