Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MPQ read speedups -- possible native code required #8

Open
collinsmith opened this issue Mar 10, 2019 · 3 comments

Comments

Projects
None yet
1 participant
@collinsmith
Copy link
Owner

commented Mar 10, 2019

I've noticed there are some slowdowns when reading files from MPQs, specifically pkexplode. I tried changing readBytes to read as many sectors at once (up to the uncompressed sector size) thinking this was one of the major issues (a compressed sector is only about 1500ish bytes, so I can read 3-4 at a time and be under 4K), but that didn't have as large of an effect as I had hoped. I also have looked into other file reading strategies, but I don't think they will provide a significant increase.

This is an issue on android where monstats.txt alone (90K compressed, 432K uncompressed) is taking 3-4 seconds to read, completely stalling the application start -- as an intermediate fix, I will likely add multi-threaded support to read the initial txt files in the background (MPQ reads are single-threaded). This will require a new AssetManager because it's only single-threaded, but I was planning on that anyways because of this exact reason -- I'd like a way to pre-load things in the background of AssetManager, since stuff like the TXTs are used extensively in-game and can be pre-loaded while the player is starting the app, selecting a character and joining a game without locking up assets that are needed ASAP. I can also implement support for bin files for at least a few of the files, since bin files are smaller and significantly faster to read.

I am considering transitioning the MPQ lib to its own module and implementing at least selective native code support for some of the more intensive operations (decompression and exploding), since those appear to be the biggest holdups, and I think handling them with native code might be something to look into.

collinsmith added a commit that referenced this issue Mar 10, 2019

Removed virtual MPQ file support
Removed virtual MPQ file support -- was for previous version where I was making fake assets
Other minor changes
See #8 on some MPQ read issues I've noticed
@collinsmith

This comment has been minimized.

Copy link
Owner Author

commented Mar 10, 2019

I quickly tried moving Riiablo.files initializer to a separate thread, and this fixed the startup slowdown, but locks are needed while required assets are loading (crashing if I reach the point where a needed asset hasn't loaded as expected). Without the separate thread, startup takes maybe 4-5 seconds on android and maybe 0.5 second or less on desktop.

Also, I have considered just extracting the excel files the first time the program runs on Android, as I think this would provide the easiest to implement/most effective speedup, and I'm already doing this on Android for the audio files, since Android has some issues reading audio input streams, but I don't want to rely on this approach because if the read speeds can be increased, I'd like to look into it to at least gauge the time sink.

@collinsmith

This comment has been minimized.

Copy link
Owner Author

commented Mar 21, 2019

I started rewriting and redesigning the mpq module using MappedByteBuffer and I'm seeing a potential ~10% speedup, but it has an awful amount of variance during testing (sometimes slower). I am also still seeing an inordinate amount of time reading monstats.txt in pkexplode.

I think I'm just about ready to try and write some native code to try and speed up the low level explode/decompress/decryption operations. I can't say if there will be much if any improvement though, I'll look for some benchmarks.

collinsmith added a commit that referenced this issue Mar 21, 2019

@collinsmith

This comment has been minimized.

Copy link
Owner Author

commented Mar 21, 2019

I added another module mpqlib that I will try and use jnigen with. I'm out of my depth here, so I think I'll have to put this on hold. The C pkexplode algorithm and jni is a bit much to deal with at once -- I'll need to convert the code to not use TCmpStruct or alter it to work with java Buffer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.