Support for Additional game format (ISO/CHD)

[komawoyo]

Some games folders have over 10,000 files in a directory. Support to scan iso and read this game format may be beneficial. Similar homebrew that supports this is webmanMOD and game managers for the PS3

[AniLeo]

You can mount the ISO and run it off the virtual drive

[Zangetsu38]

Iso not really exist for ps3 game, so if you have iso, just extract or mount

[komawoyo]

Yeah most of my games are iso (from both retail and psn versions ) to help with file transfer and organization. Transferring many small files is much slower than one big file. Also, it has to do with space saving - Referencing this link (https://www.howtogeek.com/180369/why-is-there-a-big-difference-between-size-and-size-on-disk/)

Yes, at the moment I do mount it but its not really convenient and does not utilize the QT GUI for games listing

[AniLeo]

psn versions

They need to be at dev_hdd0/game/GameID just like a real PS3, without tampering with them, you can't put them into some compressed file and then expect that having a read-only mount of it will work.

[dio-gh]

Still an "issue", also misses the longterm label.

[QuaiGoner]

It would be also very useful to have some kind of trimming/compressing tool inside of rpcs3 for ISO format *for example trimming PS3_UPDATE folder or additional languages)

[dio-gh]

That sounds like a bit of a feature creep, especially since how game localizations are packed entirely depend on the given game. At that point you could just extract the ISO, throw out whatever you want, then turn it back into an ISO. Sounds pretty out of scope in my book.

[AniLeo]

Incorrect, you don't need to write to discs since they're read only, you just need to map an ISO to dev_bdvd

Dumping games as ISO is the de facto way to dump games, Redump metadata library is also being built as both encrypted and decrypted ISOs

RPCS3 will eventually support mounting ISO, decrypted, encrypted or both. If anything, dumping to a game folder is what is wrong.

[Papermanzero]

Beside iso, i would like to add chd. Iso and chd are more practical to archive games.
So this feature is highly desirable

[Augusto7743]

Unhappily load mounted ISOs in virtual drives in Linux not work. Windows works fine.
I not has information if is an security feature in Linux not allowing rpcs3 to read all files in virtual drive.
Now if instead is implemented internally in rpcs3 read discs dumps in format CHD is an much better option.
CHD save much disk space, but has anothers details.
In Windows is possible mount dump disc in DAA format using PowerISO and another good detail is about games that need hard disk installation.
You start an game and wait the game installation. After close rpcs3 and browse to game folder and remove all folders after folder that is eboot.bin and sfo file thus is possible created hardlinks from mounted disc in virtual drive saving much more space.
I has tested and doing hardlinks works without problems since that first install the game, create the hardlinks and only after install the game updates.
In other hand if using CHD not is possible create hardlinks.
Is possible also create DAA file of PSN hard disk games and thus saving disk space. Not crash in gameplay.

In moment waiting gcdemu to be released to Ubuntu 21.04 to test with rpcs3.

[AniLeo]

The first formats that should be supported are decrypted and encrypted PS3 ISO, which are the standards for properly dumping and storing PS3 discs on containers (same for PS1 and PS2 games but only as decrypted ISO).

CHD is not used at all to store PS3 games, mostly because console CFW nor emulators can read these yet. I don't know if you can keep encrypted ISOs as CHD and then decrypt them without having to convert to ISO and back as well, but I don't see why it can't be eventually supported, even if only for decrypted games.

DAA is useless, it's just a proprietary ISO-esque format with compression. Can just use CHD or any other non-proprietary ISO-alike format that supports compression.

Also, you don't want to keep digital games on read-only containers.

[Augusto7743]

I had tested MDS, MDF and DAA when creating ISO dumps for mount in virtual drives and DAA has much better compression and for mount I had used PowerISO works well also when creating hardlinks.
If the digital game not has any new update is possible create ISOs compressed and mount in virtual drive that help to save 15 % or more disk space and also use few CPU resource.

[AniLeo]

No, directories on HDD should be writable regardless of whether the game actually writes to them or not. You cannot guarantee that a game won't try to write to storage without checking thoroughly on a game by game basis. You're not supposed to use read-only formats for them, and no one else does it regardless of emulator for that same reason.

MDS/MDF and DAA are all proprietary formats thus useless for us, so CHD is the way to go instead when it comes to compression.

[Papermanzero]

ISO and CHD is the way to go.
PCSX2 implemented CHD for iso‘s and bin‘s.
ISO is the redump format and chd should be the lossless compressed storage format

Edit:
CISO would be also possible for lossless compression

[Augusto7743]

@AniLeo
You are saying that has games write in game folder instead game data folder ?
About MDS, MDF and DAA was only to share that is possible use the 3 files formats in virtual drive and works and not waiting RPCS3 to implement read both files formats.
Exactly the 3 formats are useless to implement in RPCS3.
CHD using the latest version is the correct choice.

@Papermanzero
CISO has better compression ?
Hey that format is about PSP games ?
PS3 support boot PSP games, but is strange for me RPCS3 emulating PSP.
RPCS3 starting PSP games is strange. Users like it, but is better use an PSP emulator.

[stzokev]

I would also very much appreciate ISO/CHD format. The first is possible even now, without much hassle if you're using a front-end for your game library. Sadly there is no virtual CD/DVD/BDVD software that can work with CHD format at this time, otherwise we would have been set. I've made some tests with chd compression of PS3 ISOs and sadly the gains (600-900 MB less) were not that impressive as they are with PSX and PS2.

[Augusto7743]

"The first is possible even now, without much hassle if you're using a front-end for your game library."
How is possible ?

PS3 CHD is compressed data and PSX compress audio track in FLAC being the because have more compression ratio.

Loading PS3 games in CHD will help to use extremely less disc space and CHD not is slow to read.

[TailsGuy]

Some suffer from a disconnection from reality, everyone uses ISO. This is something essential.

Seconded.

The discussion of chd should not even have been part of this discussion at all. It's not even clear whether ps3 content can be compressed into chd, let alone would it be beneficial if it was.

The default format of ps3 games is iso. Therefore, support for iso is a necessity. Iso is also commonly used across similar consoles of the time such as ps2, gamecube, wii, xbox to an extent, xbox 360, etc. So it's just natural to desire support for ps3 iso, as iso already is the default format for ps3 game discs.

This discussion should largely revolve around iso support and how to achieve it. That should be the priority and the first

Well, we did explain why CHD would be beneficial. But yes ultimately ISO is the first priority.

[nixxou]

I wonder if the core of the issue is that the rpcs team do not want to implement a decryption routine for encrypted iso (for legal reasons) and fear that support of only unencrypted iso would lead to false report (forum, discord, github) of games not working from people that try to run encrypted one. (Well, for me, that can be solve with a popup when you try to run encrypted iso, but ...)

[AniLeo]

I'm gonna be upfront with you, the reason this is still not implemented is that for years the only activity here is theoretical discussion on compression formats or containers but no one has actually proposed a pull request to implement ISO support in any way, decrypted, encrypted, or both.

Clearly no one in the core team has any interest of working on this, otherwise they'd have picked up the ticket long ago, so it's up to anyone else to provide an implementation and open a PR, we're a FOSS project thus many features you see on the emulator today weren't necessarily implemented by the core team.

And as I already mentioned years ago, this isn't something we wouldn't accept, otherwise this ticket would have long been closed.

[nixxou]

Thanks for the clarification

[boludoz]

Some suffer from a disconnection from reality, everyone uses ISO. This is something essential.

Seconded.

The discussion of chd should not even have been part of this discussion at all. It's not even clear whether ps3 content can be compressed into chd, let alone would it be beneficial if it was.

The default format of ps3 games is iso. Therefore, support for iso is a necessity. Iso is also commonly used across similar consoles of the time such as ps2, gamecube, wii, xbox to an extent, xbox 360, etc. So it's just natural to desire support for ps3 iso, as iso already is the default format for ps3 game discs.

This discussion should largely revolve around iso support and how to achieve it. That should be the priority and the first step.

Personally I have never seen an encrypted ISO, I have been using computers for 17 years. And being a lawyer from the legal side, I tell you that no one can judge you for making a backup copy of your blueray taking into account the jurisprudence (I am a lawyer). Simply add an exception with a message if the image is not supported and that's it. Or ask those at pcsx2.

[boludoz]

I'm gonna be upfront with you, the reason this is still not implemented is that for years the only activity here is theoretical discussion on compression formats or containers but no one has actually proposed a pull request to implement ISO support in any way, decrypted, encrypted, or both.

Clearly no one in the core team has any interest of working on this, otherwise they'd have picked up the ticket long ago, so it's up to anyone else to provide an implementation and open a PR, we're a FOSS project thus many features you see on the emulator today weren't necessarily implemented by the core team.

And as I already mentioned years ago, this isn't something we wouldn't accept, otherwise this ticket would have long been closed.

Maybe when I have time I will observe how pcsx2 or some other project is implemented and I will try to implement it, but if you can start, start earlier.

[boludoz]

I'm going to see how the emulator interacts with the files and then I'll try to read an iso. I do not promise anything.

Edit:
Apparently pcsx2 uses its own ISO reader and you would have to request permission to use it, it looks very well optimized.
https://github.com/PCSX2/pcsx2/blob/master/pcsx2/CDVD/IsoReader.cpp

[Leafdomain]

Some suffer from a disconnection from reality, everyone uses ISO. This is something essential.

Seconded.
The discussion of chd should not even have been part of this discussion at all. It's not even clear whether ps3 content can be compressed into chd, let alone would it be beneficial if it was.
The default format of ps3 games is iso. Therefore, support for iso is a necessity. Iso is also commonly used across similar consoles of the time such as ps2, gamecube, wii, xbox to an extent, xbox 360, etc. So it's just natural to desire support for ps3 iso, as iso already is the default format for ps3 game discs.
This discussion should largely revolve around iso support and how to achieve it. That should be the priority and the first step.

Personally I have never seen an encrypted ISO, I have been using computers for 17 years. And being a lawyer from the legal side, I tell you that no one can judge you for making a backup copy of your blueray taking into account the jurisprudence (I am a lawyer). Simply add an exception with a message if the image is not supported and that's it. Or ask those at pcsx2.

Decrypting is the user's responsibility. When implementing ps3 iso support for rpcs3, it should just be assumed that it's a standard iso that's already been decrypted. If the executable cannot be read and is still encrypted, just throw an error and show an error message to the user.

[Leafdomain]

Decrypting is the user's responsibility. When implementing ps3 iso support for rpcs3, it should just be assumed that it's a standard iso that's already been decrypted. If the executable cannot be read and is still encrypted, just throw an error and show an error message to the user.

Extracting the game folder from iso is the user's responsibility. When implementing ps3 game support for rpcs3, it should just be assumed that it's a game folder that's already been decrypted. If the executable cannot be read and is still packed, just throw an error and show an error message to the user. 🤣

Uhm, no, allowing to read a singular rom file is a standard feature across all emulators.

[DocMAX]

My problem is i want each game in "one place". Wether it is game.iso, game.chd, game.xyz whatever extension to let game launchers launch a game with "rpcs3 game.xyz".
But this is not possible. So my current solution is a script instead like this:

I have each game in a folder. The folder contains "disc" and "hdd" subfolders. Like:
/mnt/ps3/games/Assassin's Creed II/{disc,hdd} and a serial.txt which contains the PS3 game ID.

Now i created a script which does symlink before startup:

command: startup.sh /mnt/ps3/games/Assassin's Creed II/

startup.sh:

run_ps3_game() {
    rom_path="$1"
    [ ! -d "$rom_path/hdd" ] && mkdir "$rom_path/hdd"
    rom_id=$(cat "$1"/serial.txt)
    gamename="$(basename "$(readlink -f "$1")")"
    prepare() {
        echo "Preparing PS3 game \"$gamename\""
        if [ -e "$rom_path/disc/$rom_id" ]; then
            ln -sf "$rom_path/disc/$rom_id" "$HOME/.config/rpcs3/games/"
            run_cmd="$HOME/.config/rpcs3/games/$rom_id"
        fi
        mv -T $HOME/.config/rpcs3/dev_hdd0 $HOME/.config/rpcs3/dev_hdd0.tmp
        ln -sT "$rom_path/hdd" $HOME/.config/rpcs3/dev_hdd0
        [ ! $run_cmd ] && run_cmd="$HOME/.config/rpcs3/dev_hdd0/game/$rom_id"
    }
    cleanup() {
        echo "Cleanup RPCS3 folders..."
        rm $HOME/.config/rpcs3/dev_hdd0
        mv $HOME/.config/rpcs3/dev_hdd0.tmp $HOME/.config/rpcs3/dev_hdd0
        [ -e $HOME/.config/rpcs3/games/$rom_id ] && rm $HOME/.config/rpcs3/games/$rom_id
    }
    prepare
    rpcs3 --no-gui "$run_cmd" &
    ps3_pid=$!
    echo "RPCS3 started with PID $ps3_pid"
    wait $ps3_pid
    echo "RPCS3 stopped"
    cleanup
}
run_ps3_game "$@"

What do you think of this, couldn't find a better way.

[nixxou]

What do you think of this, couldn't find a better way.

If you are on linux, i don't see why you just don't use iso, squashfs or even rar and mount/dismount pre/post rpcs3 launch.

[Leafdomain]

My problem is i want each game in "one place". Wether it is game.iso, game.chd, game.xyz whatever extension to let game launchers launch a game with "rpcs3 game.xyz". But this is not possible. So my current solution is a script instead like this

This is exactly the point. The beauty of roms is being able to organize your awesome games in singular files.

Currently I also had to make my own custom launcher that mounts iso images for ps3 games, while my launcher can run all other consoles directly with the rom and emulator.
(My recommendation for Windows users: use WinCDEmu to do the mounting.)

This is just not ideal. All major emulators support singular rom files, that's the standard.

[boludoz]

My problem is i want each game in "one place". Wether it is game.iso, game.chd, game.xyz whatever extension to let game launchers launch a game with "rpcs3 game.xyz". But this is not possible. So my current solution is a script instead like this

This is exactly the point. The beauty of roms is being able to organize your awesome games in singular files.

Currently I also had to make my own custom launcher that mounts iso images for ps3 games, while my launcher can run all other consoles directly with the rom and emulator.
(My recommendation for Windows users: use WinCDEmu to do the mounting.)

This is just not ideal. All major emulators support singular rom files, that's the standard.

I think that from Windows 7 you simply double click on the rom and it mounts as a disk, it would be very interesting to dump it into the games folder and have it work automatically.

[Leafdomain]

I think that from Windows 7 you simply double click on the rom and it mounts as a disk, it would be very interesting to dump it into the games folder and have it work automatically.

I made something that mounts the rom and starts the emulator with the game already running, with a single click, that's why I preferred WinCDEmu. :)
For other emulators it just runs the emulator and rom directly, easily. Just wishing this was already possible for rpcs3, which is why I'm contributing to the discussion here. I care a lot about the user experience.

[DocMAX]

What are you all talking about? RPCS3 games do not just have an ISO. They also have updates on hdd, which can't be merged into ISO!

[RinMaru]

Im gonna go out on a limb and say no one cares about CHD or ISO we all simply want a consolidated format so we dont have multiple folders laying around for each game.

[boludoz]

Im gonna go out on a limb and say no one cares about CHD or ISO we all simply want a consolidated format so we dont have multiple folders laying around for each game.

That is why ISO is an appropriate format, you can access the data in real time without the need for heavy decompression because it works as if it were a "hard drive".

[Leafdomain]

That is why ISO is an appropriate format, you can access the data in real time without the need for heavy decompression because it works as if it were a "hard drive".

Yup, exactly. Iso is the way to go.

[Papermanzero]

Good news.
CHD added the zstd/zarchive compression with mame version 0.262. That means all the benefits from Zarchive are now also available for CHD. Means the highest possible lossless compression of data together with dif file support and archive formats are now all avalable in chd.

so for isos as well as game folders (dlc, games) the best way to archive files is CHD.

if someone would like to implement it, this lib can be used:
https://github.com/rtissera/libchdr

[thingsiplay]

I would also love to see CHD support, but any other single file solution is highly preferable over the current frustrating way of handling the files. At least for ISO, the emulator itself does not really need to support the format in order to play games from ISO archives. Just like how its possible to mount ISO files and play from virtual drives, RPCS3 could rely on an optional third party application/driver and do it automatically.

[boludoz]

I would also love to see CHD support, but any other single file solution is highly preferable over the current frustrating way of handling the files. At least for ISO, the emulator itself does not really need to support the format in order to play games from ISO archives. Just like how its possible to mount ISO files and play from virtual drives, RPCS3 could rely on an optional third party application/driver and do it automatically.

And why don't we add everything possible?

[thingsiplay]

Because everything possible is not needed.

[Papermanzero]

Iso and chd would be a start.
I posted already a chd library to read chd files.

[birthtothunder]

Not sure how its not needed when every other relevant disc based console emulator has support for ISOs or a lossless compression format. It's a clearly wanted feature.

[ukos-git]

It's not like it is not needed. I, too, would prefer a one-file solution. MultiMan supports ripping to ISO. It is, however, still to this point not so simple to transfer large iso files from the ps3 to the pc. FAT32 is still the default for the external drive, ntfs does not work well enough and the Network speed for transfer is slow. That beeing said, repacking as a fat32 iso/chd would be an additional step to perform. I am just wondering after following the discussion for so long why nobody that wants this feature so badly is making a PR instead of complaining.

[drewbyhk]

I'd love to see this as well! I think something interesting that the devs could look into is Cemu's .WUA format, which is able to compress the game, updates, and DLC into one file. It uses ZArchive which is opensource and located here on GitHub. Seems like implementing something like that would be useful for people like me who just want one file per game rather than a folder.

[BenMcLean]

Seems like parsing ISO or CHD would be the first step to creating an emulator for any disc based system like this.

[Augusto7743]

That topic are extremely bigger. Much replies and not any action done.
Simply ... MAMEdevs are profesionals and was choiced the CHD format. Thus RPCS3 need to go in same way.
RPCS3 need add support for CHD with diff files features so allowing create game images of HDD games too. Diff is important because has HDD games does writes in game directory.
Please considering an RPCS3 dev reply above avoid reply saying the same type of message over and over again.