-
Notifications
You must be signed in to change notification settings - Fork 13k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-5300] Add more gentle file deletion procedure #2970
Conversation
I like the idea. We could extend Flink's FileSystem class to also offer something like that and then use that method. If we decide to not do that, it would be good to put the repeated logic for "delete if empty" into a utility function. |
I like the idea of not listing the status for all contained files. However, I've looked at the implementation of Hadoop's Therefore, I'll refactor the code and add a |
62d3d28
to
442cc64
Compare
I've update this PR @StephanEwen. Unfortunately, I couldn't use Hadoop's I've refactored the code to contain a method |
442cc64
to
156f68b
Compare
Rebasing on the latest master. @StephanEwen since I couldn't find a more efficient way to list the directory contents (wrt Hadoop FS) than |
156f68b
to
1f2eaa7
Compare
Looks good to me. I would actually suggest to add two tests, one in |
True. Will add the tests and then merge the PR. |
c8524a6
to
613a636
Compare
Before deleting a parent directory always check the directory whether it contains some files. If not, then try to delete the parent directory. This will give a more gentle behaviour wrt storage systems which are not instructed to delete a non-empty directory. Add test case for more gentle file deletion
613a636
to
9cc8fc1
Compare
Before deleting a parent directory always check the directory whether it contains some files. If not, then try to delete the parent directory. This will give a more gentle behaviour wrt storage systems which are not instructed to delete a non-empty directory. Add test case for more gentle file deletion This closes apache#2970.
Before deleting a parent directory always check the directory whether it contains some
files. If not, then try to delete the parent directory.
This will give a more gentle behaviour wrt storage systems which are not instructed to
delete a non-empty directory.
cc: @StefanRRichter