New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP][SPARK-40464][YARN] Support automatic data format conversion for shuffle state db #38220
Conversation
… shuffle state db
LOGGER.warn("Migrate DBBackend from {}({}) to {}({})", | ||
fromDbBackend.name(), fromFile.getCanonicalPath(), | ||
toDbBackend.name(), toFile.getCanonicalPath()); | ||
DB fromDb = initDB(fromDbBackend, fromFile, version, mapper); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should close the fromDb
and toDb
after used
Preconditions.checkNotNull(_recoveryPath, | ||
"recovery path should not be null if NM recovery is enabled"); | ||
|
||
DBBackend registeredExecutorsDbBackend = findExistedDbBackend(RECOVERY_FILE_NAME); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
line 503 ~ line 511 can be extract into a general method and put into the helper class
@@ -493,6 +496,49 @@ protected Path getRecoveryPath(String fileName) { | |||
return _recoveryPath; | |||
} | |||
|
|||
private void migrateRecoveryDb() throws IOException { | |||
Preconditions.checkNotNull(_recoveryPath, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should not be a checkNotNull
, I think it should be _recoveryPath
|
@panbingkun Any updates of this one ? |
Can one of the admins verify this patch? |
I will update it weekend. |
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.