Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload of large files via POST is out of question, unless machine has enough RAM to accommodate whole file into the memory #34

Open
xZero707 opened this issue Jul 29, 2021 · 1 comment

Comments

@xZero707
Copy link

xZero707 commented Jul 29, 2021

I planned to use this project as a backup endpoint on my Raspberry Pi but noticed that OOM happens about halfway through an upload of 4.7 GB archive (I've build an image for Arm64v8).
The memory usage peak of this container was 3.8 GB before being killed due to OOM. My Rpi has 4GB RAM.

Due to my lack of knowledge of Go (I'm learning), I cannot say whether is this This seems to be by design (File stored in memory before flushing out to the disk) or there might be a memory leak, but the RAM usage pattern doesn't support that theory well.

Opera Snapshot_2021-07-29_230326_portainer

EDIT: Found the issue (or at least one part of it): https://github.com/mayth/go-simple-upload-server/blob/master/server.go#L83
This function seems to load the whole file into the memory.

@xZero707 xZero707 changed the title Upload of large files are out of question, unless machine has enough RAM to accommodate whole file into the memory Upload of large files is out of question, unless machine has enough RAM to accommodate whole file into the memory Jul 29, 2021
@xZero707 xZero707 changed the title Upload of large files is out of question, unless machine has enough RAM to accommodate whole file into the memory Upload of large files via POST is out of question, unless machine has enough RAM to accommodate whole file into the memory Aug 1, 2021
@mayth
Copy link
Owner

mayth commented Mar 24, 2024

@xZero707 thanks for your report.
As you described, the old implementation uses io.ReadAll to read the uploaded file into memory. So this is "expected" behavior.
I've released v2 last year. In v2, I'm using io.Copy instead of io.ReadAll.
If you are still using this server, please consider using v2.
However, I didn't test such situation so this issue can be still there. If so, please mention me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants