Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add support for the WIA disc image format #8538

Open
wants to merge 18 commits into
base: master
from

Conversation

@JosJuice
Copy link
Member

JosJuice commented Dec 28, 2019

There's a lot left to be done before this can be merged, but I'm creating this PR early in case anyone is interested. Currently, you can only play games that are stored without any compression.

@JosJuice JosJuice force-pushed the JosJuice:wia branch from 2b75fb0 to a98582d Dec 28, 2019
Source/Core/DiscIO/WIABlob.h Outdated Show resolved Hide resolved
Source/Core/DiscIO/WIABlob.cpp Outdated Show resolved Hide resolved
{
WIAFileReader::WIAFileReader(File::IOFile file) : m_file(std::move(file))
{
m_file.Seek(0, SEEK_SET);

This comment has been minimized.

Copy link
@lioncash

lioncash Dec 28, 2019

Member

It seems kind of odd to put the seek into the constructor, especially given it's correcting an artifact of the current calling code that executes the constructor.

This comment has been minimized.

Copy link
@JosJuice

JosJuice Dec 28, 2019

Author Member

So you mean that it would be better to put the seek inside Create (before the call to the constructor) instead? I was thinking it would make more sense to put it inside the constructor, since that means the constructor is making less assumptions about the calling code.

@JosJuice JosJuice force-pushed the JosJuice:wia branch from a98582d to 76fed47 Dec 28, 2019
@RinMaru

This comment has been minimized.

Copy link

RinMaru commented Dec 28, 2019

never heard of this format what tool makes it? and does it have a gui or a bat file to use i have no clue how cmd works
Is this going to be added to dolphin as a compression method it can do itself?

@iwubcode

This comment has been minimized.

Copy link
Contributor

iwubcode commented Dec 28, 2019

@JosJuice - not to get too off topic but I'm curious why you chose to support this over NKit? I haven't followed either format too closely but know that many were impressed by NKit's compression ratio and apparently it even works with GCZ as well.

Regardless, looking forward to this!

@mbc07

This comment has been minimized.

Copy link
Contributor

mbc07 commented Dec 29, 2019

@iwubcode NKIT images (both .nkit.iso and .nkit.gcz formats) works out of the box with Dolphin, there's no special handling required*...

*technically Wii images in NKIT format doesn't work in 5.0 stable because it shipped with broken support for unencrypted Wii volumes, but that issue is fixed in current development builds

@iwubcode

This comment has been minimized.

Copy link
Contributor

iwubcode commented Dec 29, 2019

@mbc07 - ah ok, wasn't sure what this was doing. I didn't look at the code but yeah seeing the commits, it seems like it's being used to play compressed games.

I was thinking of more of a "right click" -> compress with -> "nkit" (or wia) and assumed @JosJuice was planning something like that but I think I misread the whole thing.

@JosJuice

This comment has been minimized.

Copy link
Member Author

JosJuice commented Dec 29, 2019

never heard of this format what tool makes it? and does it have a gui or a bat file to use i have no clue how cmd works

wit can create it. I'm not sure if there's a GUI for it.

Is this going to be added to dolphin as a compression method it can do itself?

My plan is to add the ability for Dolphin to write WIA files eventually. But I want to improve the WIA reading code more first.

@iwubcode Like mbc07 mentioned, Dolphin can already run NKit files. A design goal for NKit was to have the files be playable in Dolphin without needing to make any changes to Dolphin. Anyway, I would prefer adding writing support for WIA rather than NKit. NKit disc images are not netplay/TAS compatible with normal disc images, and there is currently a known bug with not being able to start a new game in Super Paper Mario. WIA avoids this problem (the reason NKit has these problems is the design goal I mentioned earlier), and WIA can get even better compression using bzip2 or LZMA (though that hasn't been implemented in this PR yet).

@JosJuice JosJuice force-pushed the JosJuice:wia branch from 76fed47 to 3737ddd Dec 30, 2019
@JosJuice

This comment has been minimized.

Copy link
Member Author

JosJuice commented Dec 30, 2019

I have added support for WIA's "purge" compression method. It's a simple way of storing runs of zeroes more efficiently.

The compression methods that remain unimplemented are bzip2, LZMA, and LZMA2. Unfortunately, adding support for them involves one of the parts of C++ development that I'm worst at: adding libraries. If someone wants to help me add the bzip2 and LZMA libraries to Dolphin (without necessarily changing anything in the WIA reading code), that would be much appreciated. Otherwise this is probably going to take a while.

@JosJuice JosJuice force-pushed the JosJuice:wia branch 2 times, most recently from c62c676 to 582f68a Dec 30, 2019
@RinMaru

This comment has been minimized.

Copy link

RinMaru commented Dec 31, 2019

What formats you plan to support i dont have ISO wii dumps just wbfs from usbloaderGX. will this support compressing wbfs to wia?

@JosJuice

This comment has been minimized.

Copy link
Member Author

JosJuice commented Jan 1, 2020

I currently don't have any plans to support converting from WBFS, since it would require heuristics for determining the size of the original disc image (since WBFS doesn't store the size). But maybe it could be added if enough people think it's important. For now, I would suggest using wit for converting from WBFS to WIA.

@mbc07

This comment has been minimized.

Copy link
Contributor

mbc07 commented Jan 1, 2020

@RinMaru please stay on topic. This PR discussion is not a support thread.

@Starmann

This comment has been minimized.

Copy link

Starmann commented Jan 3, 2020

I would like NKit-making support for Dolphin because IIRC, WIA doesn't work on actual hardware

@JosJuice

This comment has been minimized.

Copy link
Member Author

JosJuice commented Jan 3, 2020

Dolphin is not a program that is intended to help you run GC/Wii games on the original hardware. Also, Wii NKit doesn't work on retail consoles – only GC NKit does.

@mbc07

This comment has been minimized.

Copy link
Contributor

mbc07 commented Jan 3, 2020

And like I mentioned earlier, NKIT images work out of the box in Dolphin, no special handling is needed...

@Starmann

This comment has been minimized.

Copy link

Starmann commented Jan 3, 2020

I'm aware that Dolphin supports NKIT out of the box

@JosJuice JosJuice force-pushed the JosJuice:wia branch 2 times, most recently from 8d438f2 to 7a30287 Jan 4, 2020
@JosJuice

This comment has been minimized.

Copy link
Member Author

JosJuice commented Jan 4, 2020

bzip2, LZMA and LZMA2 now work, but there are some pretty noticeable performance problems.

@JosJuice

This comment has been minimized.

Copy link
Member Author

JosJuice commented Jan 5, 2020

I've done some speed optimizations. WIA images with a chunk size of 2 MiB are now fast enough to be playable, but I'm not sure if there's a little bit of lag when the game is loading a lot of data or if that's just my computer giving me a little bit of lag for unrelated reasons. I haven't tested larger chunk sizes, but I imagine that large chunk sizes (like wit's default, 40 MiB) are going to cause some pretty noticeable lag when the game seeks on the disc.

@JosJuice JosJuice force-pushed the JosJuice:wia branch from a1a4fe3 to c19bb97 Jan 5, 2020
@JosJuice

This comment has been minimized.

Copy link
Member Author

JosJuice commented Jan 6, 2020

I have written some documentation for the WIA format and added it to the PR, since there wasn't really any documentation for it other than source code.

@JosJuice JosJuice force-pushed the JosJuice:wia branch from 2201546 to 3b2c113 Jan 7, 2020
@JosJuice

This comment has been minimized.

Copy link
Member Author

JosJuice commented Jan 7, 2020

I've fixed a problem that made the Wii partition data reading code not handle chunk sizes larger than 2 MiB correctly. Just like I theorized earlier, there are big slowdowns when the game tries to seek if you're using wit's default chunk size of 40 MiB. Purely sequential access (like when streaming a video from the disc) runs well, though.

@JosJuice JosJuice force-pushed the JosJuice:wia branch 3 times, most recently from e6ab6c0 to dee2e8c Jan 8, 2020
@JosJuice JosJuice force-pushed the JosJuice:wia branch 2 times, most recently from 3821d88 to c87792f Jan 12, 2020
@JosJuice JosJuice force-pushed the JosJuice:wia branch from c87792f to 761090a Jan 23, 2020
@mbc07

This comment has been minimized.

Copy link
Contributor

mbc07 commented Jan 26, 2020

Maybe display an OSD message whenever the user boots a game in WIA format saying it might experience severe stuttering issues depending of the compression settings?

@JosJuice

This comment has been minimized.

Copy link
Member Author

JosJuice commented Jan 26, 2020

I was considering adding some sort of notice when using a large block size. My first thought was to put something in the game properties, but the OSD is probably a better suggestion since it's harder to avoid.

@JosJuice JosJuice force-pushed the JosJuice:wia branch 2 times, most recently from 717b6e7 to 135a8e6 Jan 26, 2020
@JosJuice JosJuice force-pushed the JosJuice:wia branch from 135a8e6 to fe56baf Jan 28, 2020
@JosJuice JosJuice force-pushed the JosJuice:wia branch from fe56baf to b4edd30 Jan 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked issues

Successfully merging this pull request may close these issues.

None yet

7 participants
You can’t perform that action at this time.