s3 sync ending prematurely

[anazar]

I'm performing the following sync

aws s3 sync s3://bucket1/files/7/2010 s3://bucket2/files/7/2010

Inside the 2010 "folder" are 12 additional folders for each month... ie (01, 02, 03 ... 12)

There are a total of about 1.5mil files spread across those 12 folders.

The sync is ending about half way through folder 09... processing only around 930k files. There are no error messages or anything.

The output is:

Completed 929024 part(s) with ... file(s) remaining

This is not isolated... I've been testing this across different directory structures w/ > 1mil files and it is ending prematurely every time.

Any thoughts?

@jamesls - have you encountered or tested on directory structures w/ 1mil+ files? Any suggestions for debugging?

Running version 1.3.5 on a medium ec2 instance.

[anazar]

additionally: running the command again seems to pick up where it left off...

seems like a big issue if it's just ending the sync prematurely w/o any notice that it didn't process all files in the sync

[anazar]

After finishing running it again it processed about 300k more files and ended up somewhere in folder 11... folder 12 never got processed.

Somehow it's either hitting a limit or it thinks there are no more files to process.

[jamesls]

I spent some time looking into this. The only thing I found was that we don't fail gracefully if we run out of inodes, otherwise I'm able to sync > 1 million files without any issues. Not sure if this is your problem, but in my case I saw:

$ df -i .
Filesystem     Inodes  IUsed IFree IUse% Mounted on
/dev/xvdb1     655360 655360     0  100% /sync

And in this situation we keep trying to sync more files despite running out of inodes. I do see the error message in the logs:

IOError: [Errno 2] No such file or directory: u'/sync/sync/74/f3/f8/6e8a3ca3f7dd32d2f6f2bdc8c11ec17a2b'

I'll make a change such that we actually stop syncing when we run out of space.

One thing you could try is to use the --dryrun flag. This will at least tell you in the CLI thinks there are more files to sync (and it's just failing to sync them) or if the CLI thinks it's synced all the files.

[jamesls]

I believe this issue should be addressed with #749 which has been fixed. Let us know if you're still having issues with the latest version of the CLI.