Sync command issue with many files.

[eminegit]

Hi, the sync command works great with a few files. I am trying to sync 2 million files into the s3 bucket which has more than 15 million files but it stops after 5 minutes with a "killed" error message. I use AWS s3 sync command; it works fine but it is just slower than the s5cmd command.

[sonmezonur]

@eminegit Hello! Could you share your s5cmd version please? We have released a new version to address this issue (v2.0.0-beta.2)

[eminegit]

Hi, the version I have is v2.0.0-beta.2-c978e4a. I used 's5cmd_2.0.0-beta.2_Linux-64bit.tar.gz' file to install s5cmd.

[misuto]

Sounds like this could be the same issue as in #441

[kucukaslan]

Sounds like this could be the same issue as in #441

I agree that this is related to it.

I've tried to reproduce error. I've tried to sync from a directory with huge number of files. I received
fatal error: runtime: out of memory not a "killed" error in Linux. The errors probably means the same thing.

I've also outlined some of the possible solutions at #441 (comment).