Samples: uploadDirectory.js, Too many open files error.

[SMKH-PRO]

Error: EMFILE, too many open files.

This error is appearing when I am uploading a directory that contains thousands of files around 4-5k+ files.

1. Is this a client library issue or a product issue?
   The issue is in the uploadDirectory sample.

2. Did someone already solve this?
   No

3. Do you have a support contract?
   No

Environment details

• OS: Google Cloud Run Container
• Node.js version: 14
• @google-cloud/storage version: 5.18.3

Steps to reproduce

1. Try to upload a directory with 5k+ file

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[ddelgrosso1]

Hi @SMKH-PRO thank you for opening this issue. I have a few followup questions for you, from the PR you linked it appears you are limiting the promises executing at any given time. When you were seeing the issue were you running all 5000+ concurrently? What were the estimated sizes of the files you were uploading?

[danielbankhead]

Thanks @SMKH-PRO, I've created an issue for parallel operations that can better suit this request: #1868

In the meantime, I've also created a separate PR to handle the 'too many open files' issue: #1867

Thank you for bringing this to our attention; we'll work on making this experience much better.

[SMKH-PRO]

Hi @ddelgrosso1 @danielbankhead ,

Yes, this error was appearing when I was uploading all the files concurrently/in parallel operation, the estimated size of each file was not exceeding 5mbs, However, this error does not appear on local machines assuming the local machine has better resources, It was occurring on the production server and this PR fixed it.

I limited the parallel operation to upload an x amount of files and tested this code and it works perfectly in the production.
the concurrencyLimit is the number of files to upload per parallel operation,

Promise.all is the parallel operation and before this PR it was uploading all 5k files in parallel, I limited it to upload an x amount of files parallel per iteration, as totally removing this parallel operation will increase upload time so instead, I limited the parallel operation so the limits do not exceed and we also save time while uploading files concurrently.

Please review #1866

Thank you

[m-yasir-khan]

We were facing the same issue, this PR fixed it for us.

[danielbankhead]

Thank you for the PR. We appreciate your time and thoughtful input - we are working to provide a built-in parallel operations to make this easier: #1868

To unblock for now, you can use the sample you provided or the updated sample via #1867 to avoid the EMFILE error.