Advice on large number of PUTs

[giacecco]

Hi All,
I need to create almost 2m small files in an S3 bucket (see the make.js script at https://github.com/Digital-Contraptions-Imaginarium/frugal-addresses-onspd/tree/write-to-s3-directly if you are curious).

I am wondering what is the best practice to use knox to do that effectively. At the moment, I am simply writing the files one by one and observing that after the first 4000 PUTs the process becomes very slow. My guess is that there is some "buffer" of PUT requests that can't get processed fast enough.

Any advice is welcome, thanks,

Giacecco

[domenic]

This could be due to any number of things. Here are my guesses:

• Versions of node before io.js did not handle HTTP connection keep-alive very well, which seems possibly relevant
• Amazon could be rate-limiting you (perhaps DDoS protection?)
• Your machine could be running out of memory or other resources

https://www.google.com/search?q=s3+batch+upload has some interesting results too