Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix slow extraction times #28

Closed
wants to merge 3 commits into from
Closed

Conversation

Xpl0itU
Copy link

@Xpl0itU Xpl0itU commented Apr 12, 2022

No description provided.

@vgmoose
Copy link
Member

vgmoose commented Nov 14, 2022

Thank you for the PR, but the root issue of the slow extraction times turned out to be for-loop-within-a-for-loop runtime complexity slowness (O(n^2) to loop through every file in the manifest, then every file in the zip, instead of O(n) to just loop through the list once).

Writeup of the issue here: fortheusers/hb-appstore#89 (comment):

The old unzipping code would then loop through this list, and then call minizip to extract the file at the path. That minizip function though is also looping through all files in the zip, creating an O(n^2) runtime. The new code builds a hash map of the files first, and then uses that as a reference to find the zipped files when going through the manifest (O(n)).

I didn't realize that the minizip method was also looping through files until closer inspection. Adding to the troubles, and the reason we mostly noticed the extreme slowdown on Wii U, was that the Wii U expects certain alignments of memory, so every single n^2 file access had additional overhead and slowness, as the library was not aligned for these sizes.

With these changes, installation time on both Wii U and Switch dropped to roughly match PC extraction times, even on the largest packages, which is a huge improvement.

@vgmoose vgmoose closed this Nov 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants