Cloud storage plugin socket usage at capacity

[TimLanzi]

Link to reproduction

No response

Describe the Bug

I am using the cloud storage plugin with an S3 adapter pointing to DigitalOcean Spaces. I have users making a lot of edits relating to images lately. After a significant amount of uploads and image swapping, I get reports of performance issues on the admin dashboard and uploads failing. I looked at the logs and I found these error messages:

@smithy/node-http-handler:WARN socket usage at capacity=50 and 102 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 114 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 120 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 121 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 125 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 135 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 145 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 156 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 158 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 161 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 163 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 166 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 169 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 170 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.

These socket capacity warnings are sometimes preceded by error messages like this:

ERROR (payload): UnknownError
    err: {
      "type": "NoSuchKey",
      "message": "UnknownError",
      "stack":
          NoSuchKey: UnknownError
              at de_NoSuchKeyRes (/home/node/app/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4809:21)
              at de_CommandError (/home/node/app/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4747:19)
              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
              at async /home/node/app/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
              at async /home/node/app/node_modules/@aws-sdk/middleware-signing/dist-cjs/index.js:225:18
              at async /home/node/app/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
              at async /home/node/app/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:173:18
              at async /home/node/app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:97:20
              at async /home/node/app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:120:14
              at async /home/node/app/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
      "name": "NoSuchKey",
      "$fault": "client",
      "$metadata": {
        "httpStatusCode": 404,
        "requestId": "[request id]",
        "attempts": 1,
        "totalRetryDelay": 0
      },
      "Code": "NoSuchKey",
      "BucketName": "bucket-name",
      "RequestId": "[request id]",
      "HostId": "[host id]"
    }

I have my Payload app hosted on a basic DigitalOcean Droplet (1 vCPU, 1GB RAM, 25GB storage, added 1GB swap space) if that helps at all.

To Reproduce

It seems like it happens when a lot of uploading is done. I haven't seen a clear-cut way to reproduce, but the best I can do is:

1. Set up a payload app with the cloud storage plugin
2. Point to a DigitalOcean Spaces bucket
3. Create an upload collection
4. Upload images in quick succession. Upload new images into existing records. Whatever can be done to trigger the errors.

Payload Version

2.14.1

Adapters and Plugins

"@payloadcms/bundler-webpack": "^1.0.6", "@payloadcms/db-mongodb": "^1.5.1", "@payloadcms/plugin-cloud-storage": "^1.1.2", "@payloadcms/plugin-nested-docs": "^1.0.12", "@payloadcms/plugin-seo": "^2.3.1", "@payloadcms/richtext-lexical": "^0.9.1", "@aws-sdk/client-s3": "^3.556.0", "@aws-sdk/lib-storage": "^3.556.0", "aws-crt": "^1.21.2"

[Stroi]

Also having this problem, after a while(not more than hour) we stop getting images and the same socket warning start showing up... Increasing the socket number didn't help.
We are hosting on render, images in S3 and DB is Mongo