Uploading many files is slow

[bkniffler]

- Do you want to request a feature or report a bug?
Feature

- What is the current behavior?
Js-Client is very slow when you need to upload >10k pages. It doesn't take advantage of parallel uploading in a single thread (say 10 files in parallel) nor does it take advantage of multiple threads (child workers). I'm pretty confident deployment could be insanely faster with these.

See: netlify/cli#318
A good example implementation of parallel and multithread work is nextjs export: https://github.com/zeit/next.js/tree/canary/packages/next/export

[bcomnes]

The client uploads 15 files at a time, which helps speed up uploading of many small files.

https://github.com/netlify/js-client/blob/27f6ce34c5d19a27268cfeac6d4a266f0d58c819/src/deploy/upload-files.js

Netlify's API has a fairly low rate limit by default though, so you may be running into that if you are experiencing slow uploads.

[bkniffler]

I've looked into the code, it is true. Forking child processes would help hugely too. If there really is rate limits kicking in while deploying, that would be a shame. Would be cool if netlify would accept single zip file deployment.

[bcomnes]

There is a Zip upload endpoint available, but there is no de-duplicated zip upload endpoint. The zip upload requires you to upload everything you want hosted for every deploy. The file manifest endpoint enables content de-duplication which is usually faster after a slower initial deploy.

One thing you may check for in your project is if every file in your site is changing or not (e.g. if you have a timestamp in your headers or something). If this is the case you end up re-uploading every file every build. If your builds deterministically regenerate unmodified pages, you can take advantage of content deduplication, where you only upload newly changed assets.