HADOOP-16885. Encryption zone file copy failure leaks temp file ._COP… #1859
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…YING_ and wrapped stream.
jojochuang
approved these changes
Feb 25, 2020
...s-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
Show resolved
Hide resolved
| private HdfsDataOutputStream safelyCreateWrappedOutputStream( | ||
| DFSOutputStream dfsos) throws IOException { | ||
| try { | ||
| return dfs.createWrappedOutputStream(dfsos, statistics); |
There was a problem hiding this comment.
In fact, it looks like HBASE-16062 is related.
|
💔 -1 overall
This message was automatically generated. |
| } finally { | ||
| if (!direct) { | ||
| deleteOnExit(target.path); | ||
| } |
There was a problem hiding this comment.
One thing I need to be confident that we're handling is the situation where the file exists at the end, create fails with an overwrite error -we don't want the deleteOnExit to suddenly delete that previous file, not if it is one we care about.
For direct writes then: we don't do that delete (good), For indirect ones, we are relying on the fact that the file being created is temporary. Because we're going to end up stamping on it aren't we?
From an S3A perspective -I don't see the codepath creating a 404 as the deleteOnExit registration takes place after the upload has succeeded: the HEAD will find the file which has just been created.
|
shell-side changes LGTM. No opinion on the HDFS code |
|
I see @jojochuang approved, so +1 I've merged it in -thanks! |
|
see #1874; I hadn't noticed the hadoop-common test failure. Apologies all; if yetus is happy I'll commit it myself |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/HADOOP-16885
Fix the leaking stream issues when access encrypted files hit exception during create.
Move the deleteOnExit to ensure the file get deleted cleanly.