Add ability to trim XCI files from UI

[amurgshere]

Enhancement

XCI files can take up a lot of unnecessary space if they are not trimmed, although there are third party tools to trim these files, most of them are clunky, difficult to use and slow to check the free space data.

This change adds a new menu item "Check and Trim XCI File" to the right click context menu of games in the library. The menu item is only enabled if the XCI file is compatible and has not already been trimmed. After selecting the menu item the user is informed how much data they can save, if the user decides to continue the Free Space will be scanned to make sure there is no data (only ASCII 0xFF characters) and the the file will be trimmed in place usign SetLength. This new functionality also uses parallel processing to process the free space data in order to ensure efficient execution, but is still limited by the speed of the disk.

The status progress bar will indicate the free space processing progress, if the operation fails it will be reported to the user, if it succeeds it will succeed silently.

NOTE: New Locale items have not been translated to other languages yet.

Tested in WIndows and MacOS (Ava), however, I am unable to test in Linux, I would welcome anyone who has the time to test this functionality on that platform.

[Vaskiemaia]

Is there the possibility of adding a button to do all available XCIs ?

[amurgshere]

Thanks for your enquiry Vaskiemaia, we could add a menu item option to process all current XCI games in the library, however, it would need a more complicated user interface I feel, as the user would want to know the progress of each file as its processed and what the outcome is, but I don't see why not. Let's see if we can get this in first and then I will look at a new menu item which can process them in bulk as a further enhancement. Cheers.

[Niwu34]

Hey,
Nice Pr - really liking the idea of saving some space :D
Went trough all my XCIs and saved around 35 gb (Most were trimmed by around 4 gb)
I booted the games and they seem to work the same way as before at the first glance (didn't do a long term test ofc)
Just one note is that the GUI gets a little funky in the status bar if you start two 'trimmings' at the same Time:
The progress bar switches inbetween the progress of the games being trimmed at that moment. Both processes get finished without any issues tho.
Tests were done on win 11 Ava build

Here some screenshots for others who are also interested:

[amurgshere]

Thanks for testing it for me Niwu34, and I know what you mean about the concurrent processing issue, we have a few options here:

1. Leave it as it is, as it is correct in that its processing multiples, but I realise it may not look that nice
2. Update the buffer size to large so it updates less often, issue will still occur but it won't look as bad
3. Stop concurrent processing and only allow one file to be processed at a time, but I think this is losing functionality
4. Make it a FIFO situation where it only updates for the earliest process until that one completes (might take little finagling)
5. Have it queue them up and prices one at a time

Happy for other suggestions?

[Niwu34]

Thanks for testing it for me Niwu34, and I know what you mean about the concurrent processing issue, we have a few options here:

1. Leave it as it is, as it is correct in that its processing multiples, but I realise it may not look that nice

2. Update the buffer size to large so it updates less often, issue will still occur but it won't look as bad

3. Stop concurrent processing and only allow one file to be processed at a time, but I think this is losing functionality

4. Make it a FIFO situation where it only updates for the earliest process until that one completes (might take little finagling)

5. Have it queue them up and prices one at a time

Happy for other suggestions?

Leaving it as it is almost seems to be the best variant for now?

With the feature @Vaskiemaia suggested you could only allow one 'manual trim' at a time.
This might be the cleanest option imo.

[Vaskiemaia]

Thanks for testing it for me Niwu34, and I know what you mean about the concurrent processing issue, we have a few options here:

1. Leave it as it is, as it is correct in that its processing multiples, but I realise it may not look that nice

2. Update the buffer size to large so it updates less often, issue will still occur but it won't look as bad

3. Stop concurrent processing and only allow one file to be processed at a time, but I think this is losing functionality

4. Make it a FIFO situation where it only updates for the earliest process until that one completes (might take little finagling)

5. Have it queue them up and prices one at a time

Happy for other suggestions?

Leaving it as it is almost seems to be the best variant for now?

With the feature @Vaskiemaia suggested you could only allow one 'manual trim' at a time. This might be the cleanest option imo.

I agree, by pressing on the Trim All it would just go through the list one by one.

[github-actions]

Download the artifacts for this pull request:

• ryujinx-Release-1.1.0+da101a4-linux_x64
• ryujinx-Release-1.1.0+da101a4-win_x64

Experimental GUI (Avalonia)

• ava-ryujinx-Release-1.1.0+da101a4-linux_x64
• ava-ryujinx-Release-1.1.0+da101a4-macos_universal
• ava-ryujinx-Release-1.1.0+da101a4-win_x64

GUI-less (SDL2)

• sdl2-ryujinx-headless-Release-1.1.0+da101a4-linux_x64
• sdl2-ryujinx-headless-Release-1.1.0+da101a4-macos_universal
• sdl2-ryujinx-headless-Release-1.1.0+da101a4-win_x64

Only for Developers

• ava-ryujinx-Debug-1.1.0+da101a4-macos_universal
• sdl2-ryujinx-headless-Debug-1.1.0+da101a4-macos_universal
• ryujinx-Debug-1.1.0+da101a4-linux_x64
• sdl2-ryujinx-headless-Debug-1.1.0+da101a4-linux_x64
• ava-ryujinx-Debug-1.1.0+da101a4-linux_x64
• ryujinx-Debug-1.1.0+da101a4-win_x64
• sdl2-ryujinx-headless-Debug-1.1.0+da101a4-win_x64
• ava-ryujinx-Debug-1.1.0+da101a4-win_x64

[gdkchan]

I don't think it is an emulator job to do any kind of modification to the ROM files, why can't users use an external tool for this and where do we draw the line? (I can see people requesting the ability to remove the update section from XCIs, convert between NSP and XCI, create compressed ROMs, merging DLCs and updates etc in the emulator if this is accepted).

although there are third party tools to trim these files, most of them are clunky, difficult to use and slow to check the free space data.

None of those are issues that can only be addressed on a emulator, someone could just create a better tool.

We have a Tools menu that would maybe be a better place for something like this. It could also have the option for bulk trimming that was suggested above (that option would not make sense as a list context menu item).

[Vaskiemaia]

I don't think it is an emulator job to do any kind of modification to the ROM files, why can't users use an external tool for this and where do we draw the line? (I can see people requesting the ability to remove the update section from XCIs, convert between NSP and XCI, create compressed ROMs, merging DLCs and updates etc in the emulator if this is accepted).

although there are third party tools to trim these files, most of them are clunky, difficult to use and slow to check the free space data.

None of those are issues that can only be addressed on a emulator, someone could just create a better tool.

We have a Tools menu that would maybe be a better place for something like this. It could also have the option for bulk trimming that was suggested above (that option would not make sense as a list context menu item).

For what it's worth a few different emulators have similar abilities including Dolphin.

Maybe it could be under tools instead of a context menu as you suggested, but I personally believe it's still something worthwhile for users

[amurgshere]

We have a Tools menu that would maybe be a better place for something like this. It could also have the option for bulk trimming that was suggested above (that option would not make sense as a list context menu item).

No problem gdkchan, thanks for the feedback, so a bulk change option in the Tools menu would be supported by yourself or do you feel it’s better to just leave it out completely?

[jduncanator]

I agree with gdkchan that this is likely better suited as an external 3rd party tool, rather than being added into the emulator itself.

Personally, the issue I have with adding functionality like this is that it's often PR'ed into the emulator, and then abandoned by the original commiter weeks or months later, leaving the core Ryujinx team to support and maintain an ever growing scope of functionality, most of which is not core to the function of the emulator itself. We end up spending more time maintaining superfluous functionality of the emulator, rather than working on the core emulation technology itself.

I think this is useful functionality, but why not bundle it into a 3rd party tool and release it? If you want to integrate it nicely with the emulator, you can easily detect the Ryujinx config, extract the ROM paths, and display a list of ROM's in your own UI for trimming.

[GamerzHell9137]

Are the trimmed files playable on other platforms like android, or is this just compatible with Ryu with the implementation? Also what about .NSP trimming? Would feel incomplete without it if it was made a feature.

[jduncanator]

Also what about .NSP trimming? Would feel incomplete without it if it was made a feature.

NSP's shouldn't need trimming, as they're eShop dumps. XCI's need trimming because they're a disk image format, and Nintendo Carts come in fixed sizes (ie. 8GB, 16GB etc). If a game is only 5GB but distributed on an 8GB cart, the XCI will be 8GB, with 3GB of "nothing" sitting in it. That's what trimming is designed to remove.

[piplup55]

it doesn't trim files with keyarea nxdumptool can trim keyarea dumps, Splatoon 3 was used for both tests i did try smb wonder just to make sure it wasn't just splatoon 3

keyarea Splatoon 3:

00:00:40.427 |N| XCIFileTrimmer Write: Opening 'filepath/Splatoon 3 [0100C2500FC20000][v0] [KA][C][NT].xci' for reading
00:00:40.427 |N| XCIFileTrimmer Write: Reading size information
00:00:40.427 |E| XCIFileTrimmer Write: The source file doesn't look like an XCI file as the header is corrupted
00:00:40.427 |N| XCIFileTrimmer Write: Closing file

keyarealess Splatoon 3:

00:08:54.728 |N| GUI.XCFileTrimmerThread XCIFileTrimmer Write: Opening 'filepath/Splatoon 3 [0100C2500FC20000][v0] [NKA][C][NT].xci' for reading
00:08:54.728 |N| GUI.XCFileTrimmerThread XCIFileTrimmer Write: Reading size information
00:08:54.728 |N| GUI.XCFileTrimmerThread XCIFileTrimmer Write: Closing file
00:08:54.728 |N| GUI.XCFileTrimmerThread XCIFileTrimmer Write: Trim Status Result: TrimSuccessful

keyarea SMB Wonder:

00:09:34.596 |N| XCIFileTrimmer Write: Opening 'filepath/Super Mario Bros. Wonder [010015100B514000][v0] [KA][C][NT].xci' for reading
00:09:34.596 |N| XCIFileTrimmer Write: Reading size information
00:09:34.596 |E| XCIFileTrimmer Write: The source file doesn't look like an XCI file as the header is corrupted
00:09:34.596 |N| XCIFileTrimmer Write: Closing file

[amurgshere]

Thank you @piplup55 for picking this up, shouldn't be difficult to fix.

[amurgshere]

Hey @piplup55 ,

I have made some code changes for this, but have not been able to test them yet as I cannot get hold of a ROM that includes the KeyArea, so am unable to test it, will keep looking to see if I can get a hold of one.

Thanks.

[piplup55]

I have made some code changes for this, but have not been able to test them yet as I cannot get hold of a ROM that includes the KeyArea, so am unable to test it, will keep looking to see if I can get a hold of one.

enable prepend key area in nxdumptool

[riperiperi]

I was able to use this to trim my dump of TOTK from around 32GB to 17.126GB. I can't really speak for piplup's issue since I don't have any dumps with key area. Do trimmers usually not read the whole XCI header for trimming?

[amurgshere]

Most of them don’t as they don’t really need to, even with the key area they still don’t need to, there is no reason they can’t read it properly though.I have updated the code to support keyarea xci files, please let me know if any issues.