Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLOUD: Update Networking::Reader to increase file upload speed #3391

merged 2 commits into from Nov 1, 2021


Copy link

@Tkachov Tkachov commented Sep 27, 2021

This only changes what's related to the local webserver (SDL_net), not to the Cloud sync (libcurl).

I revisited Networking::Reader (remembering that file upload speed wasn't as fast as I wanted it to be). This class uses a "window" buffer to detect the boundaries in POST multipart requests.

First commit does what I actually intended to do back in the GSoC days: have a circular buffer, so we don't have to shift the whole buffer all the time.

The second commit tries to avoid doing unnecessary linear string comparisons all the time by calculating a "hash" of a boundary string and dynamically recalculating "hash" of what's in buffer. Strings are only compared if these "hashes" match. This is not exactly a hash, as it's just a xor sum of all the bytes in the string/buffer. If someone knows how to implement an actual hash there -- be my guest. Simple xor sum probably gives a lot of false positives, and a better hash would decrease the amount of unnecessary checks, resulting in the increased upload speed. UPD: I've tried using a simple hash, and it not only didn't increase the speed, but also slowed it down, so it seems xorsum works just fine.

I did a couple of measurements and it seems these two changes increase file upload speed by 1.5-1.7 times.

Tkachov added 2 commits Sep 27, 2021
Instead of shifting bytes in the array every time we read 1 byte, Reader now has two "pointers": where it reads from and writes into. This makes the buffer "circular", and removes the unnecessary bytes shifting, which results in increased file upload speed (only for the LAN server; libcurl side stays intact).
Instead of doing linear strcmp-like operation after every byte read, Reader now computes a "hash" of the needle it searches for and updates "hash" of the window contents. Now, it only does strcmp if window has the same "hash" as the needle, which also improves file upload speed for ScummVM's local webserver.
Copy link

@sev- sev- commented Nov 1, 2021

Thank you!


@sev- sev- merged commit 5636ffb into scummvm:master Nov 1, 2021
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet
2 participants