[Feature Request] Clean up / compression

[Chaython]

Please do not ask for help or report compatibility regressions here, use RPCS3 Discord server or forums instead.

Quick summary

Since games "install" is there a method of deduplication between the "disc" and the "install"? There seems to be no method of deduplication or compression used. If the game is already installed, is there any possibility to delete the disc and just have a basic runner, or does install have many disc dependencies?

Details

Running MSFT Compact XPRESS 8K, on RPCS3 w/ Siren Blood Cure installed makes the total reduce from 15.8 to 10.7gb. That's a 47% decrease in size on disk.
Game is 11.1GB now 8.42GB. HDD0 4.42GB now 1.99GB.
So deduplication and compression can greatly reduce storage usage. In other benchmarks, games that are compressed load faster [HDD to Memory is generally more of a bottleneck than CPU].
Xpress 8k is a very light compression and I wouldn't expect any CPU to bottleneck load times. Unless they were already bottlenecked by the emulation itself[In my testing of running the compressed game, "system" process which is used for decompression, showed no more than 0.6% CPU utilization on my 10900f.]

1. Please describe, what part of RPCS3 would be affected by your feature:

• Gameplay
• Patches
• Installation

2. Please tell us, why your feature is important to RPCS3.

3. Please attach screenshots of the feature implemented in other projects.

4. Please provide your system configuration:

Please include.

[AniLeo]

There's nothing to deduplicate, the game disc and install data are two separated things even if they may copy files from disc to install data. You can't modify the game disc, that's a big no-no preservation wise and can corrupt the game, you can't predict what the game does with installed data.

Compression is another thing that is possible.

[RinMaru]

using windows compact on most things does a good job but not all games are compressible

[toto621-linux]

Why not using squashfs, batocera-linux have something like this on the way (HERE). I don't know if it will be efficient but at least it worth the try.

[Chaython]

The best fast transparent compression is zstd, but ntfs/windows doesn't support transparent compression of it [but most modern linux fs do]. However RPCS3, could implement zstd support.

[AniLeo]

CHD support, which is the appropriate format you want here, is already being discussed on #4021

[Augusto7743]

Today the best option is CHD

[kwinsch]

I like the idea of zSTD for all kind of internal stuff. Installs, save games etc. Would be nice if RPCS could compress these things during idle cycles.

I argue, that zSTD compressed archives are superior than CHD. CHD does not always guarantee lossless compression. It converts programmatically the input data and only safes the parts it/the programmers, knew of. With an archive, you guarantee lossless compression, since you don't touch the original iso/cue,bin etc. I have played with zSTD the last days and decompression on Android and other embedded devices is blazing fast up to level 19. After that, RAM might become an issue. Since it is so fast, even decompression before playing is not really an issue. On PC with a lot of RAM, you could even decompress the hole game direct to RAM.

[Augusto7743]

"I like the idea of zSTD for all kind of internal stuff. Installs, save games etc. Would be nice if RPCS could compress these things during idle cycles."
For Linux if the disk partition use BTRFS automatically does transparent compression faster and with better compression level than zstd.
Windows has NTFS compression not being high compression.
I had used RPCS3 in both Linux and Win ... I recommend use an Linux distro how Ubuntu or even other Ubuntu distros (have distro using less of 1 GB RAM and with optmized RAM usage for users with computer 4 or 6 GB RAM).

"I argue, that zSTD compressed archives are superior than CHD. CHD does not always guarantee lossless compression. It converts programmatically the input data and only safes the parts it/the programmers, knew of. With an archive, you guarantee lossless compression, since you don't touch the original iso/cue,bin etc."
CHD not always use lossless compression ?
In any way PS3 games files also are compressed.

"I have played with zSTD the last days and decompression on Android and other embedded devices is blazing fast up to level 19."
Really is faster, but not extremely more fast than CHD.
Linux use zstd and lzo-lre to compress zram. LZO-LRE is more fast than zstd.

"After that, RAM might become an issue. Since it is so fast, even decompression before playing is not really an issue. On PC with a lot of RAM, you could even decompress the hole game direct to RAM."
You say zstd use much RAM when decompessing data ?

CHD good detail is all files are in one container thus secure.