No copy operations performed, even when destination is completely empty #16
Labels
Comments
|
Update: The only time anything actually happens is if I specify a different prefix for the destination than in the source. Then it starts the keylister, and promptly hangs with no status indicator for a long time, and then eventually gives me a lovely stack trace: If I hit So copying is happening, but only when the destination prefix path is not the same as the source prefix path. But, this leads to another issue: if the destination prefix is different than the source prefix, the copy always happens, even if the file already exists at the destination path. |
|
Sorry - just a "me too", but I seem to be having an identical issue. I've set the input prefix to just select a small subset of the files, and this small subset definitely does not exist at the destination. This leads me to question whether it's doing something silly like using the same bucket for input and output? (despite the specified dest-bucket) |
|
Command still runs with identical output if I do something crazy like: So either it's ignoring the dest-bucket argument or I'm doing something very very stupid? (both are possibilities!) |
|
Ok - as far as I can tell, there's a regression in commit 8766708. Commit 03f547c works for me as expected. In my test files, the destination files with the specified prefix definitely do not exist. Here's the broken output And here's the working version... With regards to using a "non.existent.name" for a destination bucket (mentioned in a previous comment), the "check file exists" part of the code seems to assume that the file doesn't exist for such a case, and s3m3mirror only actually emits an error once the code attempts to actually copy the file to the non-existent bucket. |
|
Dammit. I should have tested that pull request more. This is unfortunate. I will get this fixed asap. |
|
This wasn't the pull request, which was fine. This was a stupid bug I introduced. I've fixed it, and have added a test that covers this case. As future bugs are found, each one will get a test written, to validate that the fix has worked. Closing this as fixed, but will happily re-open if there are still any problems. |
|
Awesome - many thanks :) |
|
Excellent, I'll pull it down and give it a whirl. Thanks mucho for quick response and fix! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have a backups bucket in
us-west-2that I want to mirror to another bucket inus-east-1for disaster recovery purposes. It contains ~25000 files consuming ~2TB.The key structure is:
YYYMMDD/CUSTOMER/TYPE/backup_file_nameI'm running s3s3mirror on an Ubuntu 12.04.3 LTS server, using the master branch of s3s3mirror.
I've got s3s3mirror using the credentials file from s3cmd (which I'm attempting to replace with s3s3mirror).
When I run it, here's what I get:
Nothing copied, even though there are several hundred files in the source that don't exist in the destination. Weird. I thought, perhaps it's an issue crossing regions, so I created another bucket in the same region and tried again. Same result: nothing copied, even though the destination is now a completely empty bucket.
What gives?
The text was updated successfully, but these errors were encountered: