Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
MM-7633: Optimize memory utilization during file uploads #9835
Refactored the file upload code to reduce redundant buffering and stream
Overview of changes:
Still remaining, but can be separate PRs - please let me know the preferred
Definitely separate future PRs - should I file tickets foe these?
Suggested long-term improvements - should I file separate tickets for these?
Thanks @levb for the pull request!
Please help complete the Mattermost contribution license agreement?
This is a standard procedure for many open source projects. Your form should be processed within 24 hours and reviewers for your pull request will be able to proceed.
Please let us know if you have any questions.
We are very happy to have you join our growing community! If you're not yet a member, please consider joining our Contributors community channel to meet other contributors and discuss new opportunities with the core team.
4 times, most recently
Nov 16, 2018
crspeller left a comment
Overall this is a great step forward for optimizing file uploads. It adds quite a bit of complexity, but I think it's handed decently.
A follow up ticket for the webapp and RN app to make sure they are using the best method would be great. We should also think about removing the fallback code if we move to APIv5 as it does add a bit of complexity.
Do you mean that the image dimensions match the original orientation and not the dimensions of the image after we've made it upright? Like if you have an image with a width of 200 and a height of 300 that is rotated by 90 degress, it still saves width = 200 and height = 300 even though the file on disk has those dimensions flipped? If that's the case, I think that would be a bug
@hmhealey It is my understanding that we upload the original image "as is", we do not flip it since it already contains the necessary metadata, and all user agents respect it. However, if the image was flipped, we seem to be recording its dimensions in FileInfo to match the actual display.
I figured that perhaps we were using the FileInfo dimensions for sizing UI elements before the image is downloaded, so they had to be "real"? If you can point me at where/how they are used, maybe we can add a test from that?
Maybe file a ticket to look into correcting that so we can spend some more time thinking about it? Perhaps we could add an orientation field to the FileInfo and make it so that the width/height always match the native values from the image, although that would have to be done with enough time to ensure that the mobile app is backwards compatible with those changes (mostly for the case of an old app with a new server)
@hmhealey I think we should leave it as is for now, and I'll add a test that documents/verifies the behavior. When we look into cleaning up image processing performance issues, we can look into maybe adding the thumbnail/preview sizes to FileInfo explicitly, that way the client doesn't need to guess-calculate them.
The other high level comment I wanted to add is that: is there any way to simplify the assumptions? There's a good deal of complexity here to preserve 100% compatibility with the existing API, but what if we exposed a new API with tighter semantics: no
This would mean maintaining the old API until the next major version upgrade (and coordinating with the mobile app around this), but maybe it would be worth it?
@lieut-data I appreciate the advantages of simplifying the new API by keeping the backwards compatibility out of it. The main reason I decided to include the "legacy" path in the new implementation so that we can cleanly switch the various file upload use-cases to the new API and replace all of the old code. It was also nice for testing not to have 2 separate end-points to test.
With the changes you hinted at, I replaced the obscure request hacking with a couple small functions to make our own