Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream File Uploads Directly to Disk to Prevent Memory Overload #5

Closed
ahmadnasriya opened this issue Aug 20, 2024 · 0 comments · Fixed by #8
Closed

Stream File Uploads Directly to Disk to Prevent Memory Overload #5

ahmadnasriya opened this issue Aug 20, 2024 · 0 comments · Fixed by #8
Assignees
Labels
critical High-priority issues that need immediate attention. enhancement New feature or request memory management Improvements related to managing memory usage effectively. performance Issues or enhancements related to optimizing performance.

Comments

@ahmadnasriya
Copy link
Member

ahmadnasriya commented Aug 20, 2024

Currently, the bodyParser function accumulates the entire request body into memory, which can cause significant memory usage, especially when handling large file uploads. This can lead to performance degradation and potential server crashes.

Task:

Refactor the file upload handling process to stream files directly to a temporary location on disk as they are being uploaded. Once the upload is complete, move the file to its final destination.

Steps to Implement:

  1. Modify the request handler to stream the incoming file data directly to disk, rather than accumulating it in memory.
  2. Ensure that the stream handles errors gracefully and that partially uploaded files are cleaned up.
  3. Implement a function to move the file from the temporary location to the final destination once the upload is complete.
  4. Update existing unit tests and add new tests to cover the streaming and file-moving logic.

Expected Outcome:

The server should handle file uploads more efficiently, preventing memory overload and maintaining server performance.

@ahmadnasriya ahmadnasriya added enhancement New feature or request performance Issues or enhancements related to optimizing performance. memory management Improvements related to managing memory usage effectively. critical High-priority issues that need immediate attention. labels Aug 20, 2024
@ahmadnasriya ahmadnasriya self-assigned this Aug 20, 2024
ahmadnasriya added a commit that referenced this issue Aug 24, 2024
@nasriyasoftware nasriyasoftware locked as resolved and limited conversation to collaborators Aug 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
critical High-priority issues that need immediate attention. enhancement New feature or request memory management Improvements related to managing memory usage effectively. performance Issues or enhancements related to optimizing performance.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant