Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-17547] Ensure temp shuffle data file is cleaned up after error #15104

Conversation

@JoshRosen
Copy link
Contributor

JoshRosen commented Sep 14, 2016

SPARK-8029 (#9610) modified shuffle writers to first stage their data to a temporary file in the same directory as the final destination file and then to atomically rename this temporary file at the end of the write job. However, this change introduced the potential for the temporary output file to be leaked if an exception occurs during the write because the shuffle writers' existing error cleanup code doesn't handle deletion of the temp file.

This patch avoids this potential cause of disk-space leaks by adding finally blocks to ensure that temp files are always deleted if they haven't been renamed.

@JoshRosen

This comment has been minimized.

Copy link
Contributor Author

JoshRosen commented Sep 14, 2016

When reviewing this change, it may help to look at the whitespace-free diff: https://github.com/apache/spark/pull/15104/files?w=0

@SparkQA

This comment has been minimized.

Copy link

SparkQA commented Sep 15, 2016

Test build #65403 has finished for PR 15104 at commit 82bceb3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@rxin

This comment has been minimized.

Copy link
Contributor

rxin commented Sep 15, 2016

@zsxwing

This comment has been minimized.

Copy link
Member

zsxwing commented Sep 15, 2016

LGTM

@JoshRosen

This comment has been minimized.

Copy link
Contributor Author

JoshRosen commented Sep 15, 2016

I'm going to merge this to master and branch-2.0 and I'll also see whether it can be cleanly cherry-picked to branch-1.6.

asfgit pushed a commit that referenced this pull request Sep 15, 2016
SPARK-8029 (#9610) modified shuffle writers to first stage their data to a temporary file in the same directory as the final destination file and then to atomically rename this temporary file at the end of the write job. However, this change introduced the potential for the temporary output file to be leaked if an exception occurs during the write because the shuffle writers' existing error cleanup code doesn't handle deletion of the temp file.

This patch avoids this potential cause of disk-space leaks by adding `finally` blocks to ensure that temp files are always deleted if they haven't been renamed.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #15104 from JoshRosen/cleanup-tmp-data-file-in-shuffle-writer.

(cherry picked from commit 5b8f737)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
asfgit pushed a commit that referenced this pull request Sep 15, 2016
SPARK-8029 (#9610) modified shuffle writers to first stage their data to a temporary file in the same directory as the final destination file and then to atomically rename this temporary file at the end of the write job. However, this change introduced the potential for the temporary output file to be leaked if an exception occurs during the write because the shuffle writers' existing error cleanup code doesn't handle deletion of the temp file.

This patch avoids this potential cause of disk-space leaks by adding `finally` blocks to ensure that temp files are always deleted if they haven't been renamed.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #15104 from JoshRosen/cleanup-tmp-data-file-in-shuffle-writer.

(cherry picked from commit 5b8f737)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
@asfgit asfgit closed this in 5b8f737 Sep 15, 2016
@JoshRosen JoshRosen deleted the JoshRosen:cleanup-tmp-data-file-in-shuffle-writer branch Sep 15, 2016
zzcclp added a commit to zzcclp/spark that referenced this pull request Sep 19, 2016
SPARK-8029 (apache#9610) modified shuffle writers to first stage their data to a temporary file in the same directory as the final destination file and then to atomically rename this temporary file at the end of the write job. However, this change introduced the potential for the temporary output file to be leaked if an exception occurs during the write because the shuffle writers' existing error cleanup code doesn't handle deletion of the temp file.

This patch avoids this potential cause of disk-space leaks by adding `finally` blocks to ensure that temp files are always deleted if they haven't been renamed.

Author: Josh Rosen <joshrosen@databricks.com>

Closes apache#15104 from JoshRosen/cleanup-tmp-data-file-in-shuffle-writer.

(cherry picked from commit 5b8f737)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
(cherry picked from commit 8646b84)
wgtmac added a commit to wgtmac/spark that referenced this pull request Sep 19, 2016
SPARK-8029 (apache#9610) modified shuffle writers to first stage their data to a temporary file in the same directory as the final destination file and then to atomically rename this temporary file at the end of the write job. However, this change introduced the potential for the temporary output file to be leaked if an exception occurs during the write because the shuffle writers' existing error cleanup code doesn't handle deletion of the temp file.

This patch avoids this potential cause of disk-space leaks by adding `finally` blocks to ensure that temp files are always deleted if they haven't been renamed.

Author: Josh Rosen <joshrosen@databricks.com>

Closes apache#15104 from JoshRosen/cleanup-tmp-data-file-in-shuffle-writer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.