s3cmd sync doesn't detect remote files

[obecker]

Every time I'm running a s3cmd sync . s3://www.***** --delete-removed --acl-public command to sync my local files to a S3 bucket, it will upload all of my local files. Using the `--verbose`` flag outputs the following

INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 91 files, this may take some time...
INFO: Retrieving list of remote files for s3://www.*****/. ...
INFO: Found 91 local files, 0 remote files
INFO: Verifying attributes...
INFO: Summary: 49 local files to upload, 42 files to remote copy, 0 remote files to delete

I can run the command a second time and it will give the same info again and it will upload again all of the files.

Why does it say 0 remote files?
I'm pretty sure this has worked properly in a previous version of s3cmd.

[frankcrawford]

I'm also seeing this since upgrading from 1.5.1 to 1.6.0. I'm just running s3cmd sync . s3://my-bucket/ and it has suddenly started trying to upload all the files rather than just the changed files.
Running -v -d gives:

INFO: Retrieving list of remote files for s3://my-bucket/. ...
INFO: Forwarding request to ap-southeast-1
INFO: Found 290799 local files, 0 remote files
INFO: Verifying attributes...
INFO: Summary: 283556 local files to upload, 7243 files to remote copy, 0 remote files to delete

Some additional information I just noticed. I rolled back to 1.5.1 and it worked fine. However, I did notice that there was a very long pause between the request for the remote files and the report, as you would expect, but for 1.6.0, the response was almost immediate. This leads me to think that the issue is that in 1.6.0 it does not get the remote list of files for some reason.

[frankcrawford]

Okay, after some testing, I've found what the issue is, although I don't know what the difference is in the code. It appears that 1.6 treats directories differently if the end in '/' or not. So for my earlier issue:

s3cmd sync . s3://my-bucket/

doesn't match anything, but

s3cmd sync ./ s3://my-bucket/

works as previously, and matches fine.

In fact, now looking at the messages the issue makes sense, if appears that if the local source does not end in a '/' it is added to the destination bucket, but when it adds the lone `.' that is not a valid path in S3, hence no matches.

This appears to be different than for 1.5.1, where it didn't add the lone `.' or in fact any of the local source.

[fviard]

@frankcrawford Thank you for your investigation. Is it possible for you to try the current master version of s3cmd? I did modify a lot of things related to the handling of "/".

[frankcrawford]

Okay, I'll give it a go in the next day or two.
Frank
On Sat, 2015-11-14 at 10:24 -0800, Florent Viard wrote:

@frankcrawford Thank you for your investigation. Is it possible for
you to try the current master version of s3cmd? I did modify a lot of
things related to the handling of "/".
—
Reply to this email directly or view it on GitHub.

[obecker]

Yes, adding a slash works for me too. Thanks a lot.

[frankcrawford]

@fviard sorry for the delay. I downloaded the latest version from version from here, and unfortunately, there was no difference, it still fails to match the directory if it is given as either "." or ".../."

A quick look at the code shows that while there is special coding for ending in "/", there is nothing about "." or "/.". I think it would probably be an easy fix (i.e. delete a trailing '.') I'm not sure I would find all the spots that may need it.

[ffernand]

Please note that it also doesn't work (for 1.6.0) if we're syncing just one file (no directory reference).

ie) s3cmd sync /path/to/file.txt s3://bucket/aa/bb/sha1_of_filename

I've had to revert to 1.5.2 to get this to work again.

[ab]

👍

[thoys]

👍 got same bug. changing the local directory to ./ instead of . helped thanks @frankcrawford !

[felixbuenemann]

Same problem and the solution of ./ works. This should really be fixed, because it works quite different from other unix utilities.

[jrmiller82]

+1 Just encountered this issue with s3cmd (Debian testing) ./ worked and . did not match.