Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API doing a ReadAll() on input is not reliable due to memory constraints #187

Closed
phish3y opened this issue Jun 2, 2022 · 4 comments · Fixed by #202
Closed

API doing a ReadAll() on input is not reliable due to memory constraints #187

phish3y opened this issue Jun 2, 2022 · 4 comments · Fixed by #202
Assignees
Labels
bug Something isn't working

Comments

@phish3y
Copy link
Contributor

phish3y commented Jun 2, 2022

No description provided.

@phish3y phish3y added the bug Something isn't working label Jun 2, 2022
@frantjc
Copy link
Contributor

frantjc commented Jun 13, 2022

we pretty much need to just take their content-type header and assume they're correct about what they're sending, right? no more validation.

@frantjc
Copy link
Contributor

frantjc commented Jun 13, 2022

imo this would speed up upload times a lot, I have a feeling uploads are slow due to resource constraint differences between our cluster and our monsterous workstations with nothing else running on them

@phish3y
Copy link
Contributor Author

phish3y commented Jun 13, 2022

@frantjc I'm willing to drop the content type verification in the API layer to support better performance and reliability. Let it fail in the task if the input is BS

@phish3y phish3y assigned phish3y and frantjc and unassigned phish3y Jun 13, 2022
@frantjc
Copy link
Contributor

frantjc commented Jun 13, 2022

There's also some code in objectstore/s3_volume.go that is not properly implementing io.Reader which would save on memory when streaming big files from s3 I think.

@frantjc frantjc mentioned this issue Jun 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants