-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix S3.rename
didn't correctly remove the old file
#248
Conversation
/test |
Successfully created a job for commit 83a36ac: |
1 similar comment
Successfully created a job for commit 83a36ac: |
83a36ac
to
18c64d1
Compare
/test |
Successfully created a job for commit 18c64d1: |
1 similar comment
Successfully created a job for commit 18c64d1: |
/test |
Successfully created a job for commit 7e5074d: |
1 similar comment
Successfully created a job for commit 7e5074d: |
7e5074d
to
e5d55ff
Compare
/test |
Successfully created a job for commit e5d55ff: |
1 similar comment
Successfully created a job for commit e5d55ff: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome!
In the current pfio,
S3.rename
just copies the content into a new file, not removing the original one.Fixing
rename
This is because that
rename
callsremove
to delete the source object with the key normalized, which is then normalized again inside theremove
. Soremove
tries to remove the specified key that does not exist instead of the original source one that we actually need to delete.https://github.com/pfnet/pfio/blob/2.0.1/pfio/v2/s3.py#L523
This PR fixes the unnecessary normalization when calling
remove
inrename
.Fixing
remove
One of the reasons why the incorrect
rename
behavior is missed would be becauseremove
didn't raise anything even if the specified target doesn't actually exist.This S3 behavior is inconsistent to other filesystems.
Since it seems to be a constraint comes from boto3 or perhaps S3 itself, I added an explicit existence check.
Test refactoring
In addition, since S3 test cases it getting increased, I DRYed fixtures to make each test case a little simpler.
The following conventional test case
can be equivalently rewritten as: