DolphinTool: Add CLI tool subsystem + commands for verifying and converting RVZ/ISO/WIA/GCZ

[ssdsnake]

Preface:
Still having trouble with Windows. May reach out for help
Evolution of #10246

Use case:
Adds a command line "tool" project and shell. Initial commands include (a) ability to verify and output SHA1/MD5/CRC32 digests of RVZ / ISO / NKit and (b) ability to convert between RVZ/ISO/WIA/GCZ formats; both using the same routines as the GUI verifier widget.

Usage and Examples
Basic Subcommand Shell

$ ./dolphin-tool 
usage: tool COMMAND -h

commands supported: [convert, verify]

Verify Mode Usage

$ ./dolphin-tool verify -h
Usage:  verify [options]...

Options:
  -h, --help            show this help message and exit
  -i FILE, --input=FILE
                        Path to disc image FILE.
  -a ALGORITHM, --algorithm=ALGORITHM
                        Optional. Compute and print the digest using the
                        selected algorithm, then exit. [crc32|md5|sha1]

Full Verification

$ ./dolphin-tool verify -i /tmp/test.rvz
CRC32: dcd8fa3d
MD5: 03c6f44267aa4a5c42ab20698f88f477
SHA1: f2439bbe1ff64133050fbc00574be8478210a958
Problems Found: No

Simple Verification

$ ./dolphin-tool verify -i /tmp/test.rvz -a sha1
f2439bbe1ff64133050fbc00574be8478210a958

Convert Mode Usage

$ ./dolphin-tool convert -h
Usage: convert [options]... [FILE]...

Options:
  -h, --help            show this help message and exit
  -i FILE, --input=FILE
                        Path to disc image FILE.
  -o FILE, --output=FILE
                        Path to the destination FILE.
  -f FORMAT, --format=FORMAT
                        Container format to use. Default is RVZ. [iso|gcz|wia|rvz]
  -s, --scrub           Scrub junk data as part of conversion.
  -b BLOCK_SIZE, --block_size=BLOCK_SIZE
                        Block size for GCZ/WIA/RVZ formats, as an integer. RVZ
                        suggested: 131072 (128KiB)
  -c COMPRESSION, --compression=COMPRESSION
                        Compression method to use when converting to WIA/RVZ.
                        Suggested: Zstandard (zstd). [none|zstd|bzip|lzma|lzma2]
  -l COMPRESSION_LEVEL, --compression_level=COMPRESSION_LEVEL
                        Level of compression for the selected method, if it is
                        not 'none'. 5 is suggested for 'zstd'.)

Convert Mode Example

$ ./dolphin-tool convert -i /tmp/test.rvz -o /tmp/test_proof.rvz -f rvz -b 131072 -c zstd -l 22
Success

$ sha1sum /tmp/*.rvz
852abeb852867748ba451016523fd28163622c79  /tmp/test_proof.rvz
852abeb852867748ba451016523fd28163622c79  /tmp/test.rvz

[phire]

I'm happy with this design.

My only minor nitpick is that I think dolphin-tool is a better command name than dolphin-emu-tool. But I'm not going to fight for it.

[JosJuice]

In general, my old comments have been addressed now. But it seems like there's no longer anything to prevent you from using block sizes that don't work at all. For instance, I'm not sure if GCZ supports block sizes that aren't a power of two (maybe it does? Better check), and RVZ doesn't support block sizes smaller than 2 MiB that aren't a power of two, doesn't support block sizes smaller than 32 KiB at all, and (just like WIA) requires block sizes larger than 2 MiB to be an integer multiple of 2 MiB. This is particularly important to get right, because we never want to end up in a situation where it looks like Dolphin converted a file successfully but the file is actually unreadable.

[JosJuice]

LGTM if you squash the commits.

[shuffle2]

is there a reason to have this as a separate binary instead of adding the commandline options to a common location? It seems both DolphinQt and DolphinNoGUI could house this functionality with no need for separate tool...?

[ssdsnake]

They serve different purposes and bring up larger environments (graphics subsystem, etc). I didn't want to clutter those use cases or risk breaking CLI workflows for a poweruser tool.

[TomTurbine]

Hate to necro this but seems the most appropriate place to ask this.

When it comes to verifying your games, it is possible to do so against the ReDump datafiles without having to convert them from RVZ back to ISO?

And if possible, any way to include this feature in a future release of the Dolphin tool? Would gladly use another program to edit a dat file to what I need and then load it into your software if it allowed users to audit their collections against it.

[Pokechu22]

For RVZ, yes, you can verify without converting (it internally converts back while verifying, the same way it internally converts back when the game tries to read data for the disc). For other formats I'm not sure whether the same guarantee applies or not (it'd depend on whether it was scrubbed for instance).

[TomTurbine]

I understand it can verify and give you the check sum values, but I can't find a rom manager that supports using it except 1 which seemed to give me issues using it.

Was asking if there would be any way to have it verify the games against a dat file like from ReDump or potentially build in support for doing so in the future?

Basically a process of load in the dat file that you either just got or customized yourself to match your games, load in your games and have it run them to verify them. Will show you any that are missing or corrupted.

Edit: But thank you for the very swift reply. Was expecting it to take hours or days if it got answered at all.

Edit again: Also learned the hard way just how kids can destroy the integrity of a hard drive by thinking it would be "funny" to play with your fuse box without warning you.

[mbc07]

Dolphin can verify any supported ISO format against the Redump DATs, but on the GUI you can only do this manually, one by one. Exposing this feature via DolphinTool would likely allow batch verifying many ISOs with some basic CLI automation, but AFAICT that's not possible at the moment...

[TomTurbine]

Thank you for the reply.

Maybe this could be a good future feature to add but given all that just happened with steam, I am figuring this will be low on the priority lists if ever on them.

Would be nice to just load a dat file and have it check them for you. I found another tool named "OxyMoron" or something like that that claims to support it but I just couldn't seem to get it going and the other program I found extracted it to a hard drive in ISO then scanned it and then deleted it which can seem excessive to audit.

[AdmiralCurtiss]

You should be able to get the hash out of DolphinTool and then forward/pipe that into something that checks it against the dat, eg.

>dolphintool verify -i "e:\Game Images\Wii\RPBP01.rvz" -a sha1
8ae47b225b154f11d29e8e39aceb2f037739f430

-> http://redump.org/disc/50595/

If you need something more fancy than that then you can always open a PR or add a feature request on our bug tracker.

[alucryd]

@TomTurbine oxyromon author here, RVZ should be working fine (it does here), feel free to open an issue if you're having issues with it.

[TomTurbine]

Thank you Alucryd, I am good for now, but will check back into it again later.

I think my biggest issues I had was mostly the use of command-line. I used to be pretty good at the stuff back in the 90s and early 2000's but that skill set is pretty rusty nowadays.

[alucryd]

No problem. I'm slowly making the web interface more than just for visualization, the long term goal is to bring it on par with the CLI.