plugin-cloud-storage: OOMKill Error with GCS Adapter when Uploading Large Files (> 2GB) in Production

[beewhoo]

When using the GCS (Google Cloud Storage) adapter in our production environment, we are experiencing Out-of-Memory (OOM) kill errors when attempting to upload large files greater than 2GB. Our server has 2GB of RAM, and it seems that the GCS adapter is not handling memory efficiently for such large file uploads.

It appears that similar issues with large file uploads and memory management have been addressed for the S3 adapter in issue payloadcms/plugin-cloud-storage#37 and the Azure adapter in issue payloadcms/plugin-cloud-storage#46. However, the GCS adapter does not seem to have a corresponding fix or enhancement for this specific problem.

Kindly request assistance from the development team to investigate and address this issue with the GCS adapter. It is causing disruptions in our production environment and preventing us from successfully uploading files larger than 2GB.

@denolfe
@DanRibbens

[denolfe]

Thanks for the report @beewhoo . I'll need to do some digging on what's available with GCP for streaming from disk without loading the file into memory.

[josephjorgensen]

@denolfe have you had a chance to look at this yet? We're very interested as this is the most memory-hungry part of our app right now.

[denolfe]

I have not had a chance, but this does seem possible according to the example here. Open to accepting a PR.

[denolfe]

@josephjorgensen Thought about this a bit more. Where are you deploying this? Do you have the ability to write to disk? A half-measure may also be to support the useTempFiles option like the others.

Let me know your thoughts.

[josephjorgensen]

@josephjorgensen Thought about this a bit more. Where are you deploying this? Do you have the ability to write to disk? A half-measure may also be to support the useTempFiles option like the others.

Let me know your thoughts.

We're deploying via Render.com. I can look into the useTempFiles option and the pipe example your linked to above, those seem like good options.

[paulpopus]

This plugin is now being maintained in the Packages Directory of the Payload Monorepo. This repo will soon be archived and all open issues will be closed. This issue, however, will be transferred over.