Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Request] Support for CHD format #10417

Closed
Danixu opened this issue Dec 16, 2017 · 142 comments · Fixed by #18198
Closed

[Request] Support for CHD format #10417

Danixu opened this issue Dec 16, 2017 · 142 comments · Fixed by #18198

Comments

@Danixu
Copy link

Danixu commented Dec 16, 2017

Hello,

I'm using from about a month this emulator, and I've discovered the CHD format. It uses lzma compression so is a space saver (much more than CSO format). Now i'm storing the games in 7z and extracting the ISO when I want to play, but CHD has both benefits (stored in lzma format and playable).

Please, add support for CHD to this core.

Thanks!!

@RinMaru
Copy link

RinMaru commented Dec 16, 2017

adding it to the standalone version would also be nice since not everyone uses RetroArch

@hrydgard
Copy link
Owner

We already support CSO format. While it might compress slightly worse, it's still pretty good.

@unknownbrackets
Copy link
Collaborator

unknownbrackets commented Dec 17, 2017

All such formats will compress files in blocks, which won't compress as well as a 7z of the entire file. Have you actually compared a CHD of an ISO to a CSO?

Note that you can use maxcso to create CSOs with a larger block size, and using 7-zip's deflate algorithm, which compresses better than most CSO programs. Files compressed this way already work fine in PPSSPP.

-[Unknown]

@i30817
Copy link

i30817 commented Dec 20, 2017

One 'advantage' of CHD is that chdman and mame are supposed to have support for a form softpatching (parent-clones relationships, where clones store different files/sectors).

Pity it doesn't work anywhere else (including retroarch, except, ofc, the MAME core)... and they nixed my proposal on their chdman issue report page to add 'reversible hardpatch' to their format (basically, do the boring work of integrating a 'patch update' tool that takes a binary patcher, patches the underlying bytes, and creates a reversal patch to add as header for the next 'patch update' and recompress that again).

Big advantage of such schemes is that they don't depend on lazy/dead emulators implementing softpatching to work (reimplementing the wheel every time), while still being easy to update. Emulators still need to implement reading chd, but that's much easier. The devs thought that since the format specifies that the format should support parent/clone relationships this wasn't needed. I'm more pessimistic, complicated features are the first to be handwaved away.

Anyway, CHD is bad for now for retroarch anyway because the metadata database and their scanner is not ready to accept CHD checksums, internal or external (even the internal checksums are different from the raw cue/bin because they're of a 'sum' of the involved files, not only the 'first track' or whatever hack RA is using - they previously used cues and gdi and of course this failed alot since many gdi are indistinguishable due to dumping group bad idea, so i'd actually prefer the CHD approach). This may change thou, since they seem proud of the CHD support.

@ppmeis
Copy link
Contributor

ppmeis commented Dec 20, 2017

@unknownbrackets I know the question is not for this topic, But do you have a plan to make a GUI for maxcso? What I want to do is to compress iso files, but keep integrity. I mean:

1st - Original ISO file, check CRC32 on database (for example I use renascene)
2nd - Compress ISO file with maxcso (looking for zso because as I read before is the best format in terms of compression and load times.
3rd - Uncompress ZSO file and the obtained ISO file has same CRC32 than before.

With CSOPlus you will have the best compression but that's because it will delete UPDATE partition, so ISO file will never be as the original once you decompress it.

@LunaMoo
Copy link
Collaborator

LunaMoo commented Dec 20, 2017

Is ZSO even ment for the biggest compression?
From my test with available formats from best compression to worst:
PBP -> CSO2 with bigger block size -> CSO -> ZSO

Using The Leecherman's "ISO~PBP converter" for PBP and maxcso for other formats, maybe I just don't understand the settings which I should use for ZSO, but from my testing it was the fastest while leaving the biggest file.

Anyway if few mb's is a big deal PBP format seems the best for me, it's also supported on PSP and doesn't modify the file in any way, commonly overlooked with some myths caused by Sony's using it for non-PSP games as well, but PPSSPP supports it just fine.

CHD did had a sligly better compression even than PBP, but I don't think it's worth storing PSP games in that format, it's not native to the platform and doesn't have anything which would make it worth using while coming with all MAME stuff that we don't care about.

Guess I'll change the title as PPSSPP is not related to RA it looks like posted in wrong place.

@LunaMoo LunaMoo changed the title [Request] Support for CHD format in RetroArch [Request] Support for CHD format Dec 20, 2017
@unknownbrackets
Copy link
Collaborator

unknownbrackets commented Dec 20, 2017

The performance difference for ZSO only really matters on the 222Mhz CPU of the PSP. A CFW implemented it, so I tried experimenting with it, but it really didn't seem all that worthwhile for desktop.

If your device has a Mhz higher than 1000, it's probably going to make almost no difference in speed (for decompression.) And CSO compresses better.

However, LZMA has a more significant impact typically on CPU, mostly because it usually uses larger lookbehind buffers and lots more memory and memory searching. This means using CHD might have a non-negligible performance impact compared to using CSO/ZSO even for modern devices. Would need to be tested, though.

Not really planning to add a GUI to maxcso, and also not planning to make it do destructive changes.

Note that maxcso is kinda like pngcrush - it tries to compress the same data multiple times to achieve the very best compression ratio. It will usually compress slower than other tools (although can use tons of cores, so not always), but get a better ratio.

-[Unknown]

@RinMaru
Copy link

RinMaru commented Jan 18, 2020

there any update on this?

@hrydgard
Copy link
Owner

No.

@lamvuong2019
Copy link

Sorry to ask, but it seem that this hasn't been looked fior a while now. Do you thing its possible at all?

@hrydgard
Copy link
Owner

How much of a win is CHD over CSO anyway? CSO already successfully squishes games pretty well, as previously mentioned, and we do support that.

@LunaMoo
Copy link
Collaborator

LunaMoo commented Apr 28, 2020

Depends on data, but if I recall maybe alike 10mb per gb, PBP container which we also support is somewhere between max compression CSO and CHD, making this pretty irrelevant, through there's only windows software for PBP, so CSO is more widely used.

@lamvuong2019
Copy link

I will run a test now on Final Fantasy type 0 and will come back to you with the result for chd, gz and cso level 9

@Sanaki
Copy link

Sanaki commented Apr 28, 2020

I did a quick check on two games:

Dissidia 012 - Duodecim Final Fantasy (USA).iso  1674M
Dissidia 012 - Duodecim Final Fantasy (USA).cso  1291M
Dissidia 012 - Duodecim Final Fantasy (USA).chd  1154M
Patapon (USA).iso  326M
Patapon (USA).cso  211M
Patapon (USA).chd  161M

In theory, chd should also have faster I/O, but I don't have any easy way available to benchmark that.

@Tuxie
Copy link

Tuxie commented Apr 28, 2020

The biggest win with CHD support is being able to use the same tool everywhere, and native fast scan support in dat tools like clrmamepro. The disk space savings are nice too.

@lamvuong2019
Copy link

Isn't PBP used for psx iso only?

I don't know if this is a good example or not

Final Fantasy Type-0 (English Patched v2)
Original 3084866
CSO Failed
CHD 2410963 - 22%
GZ 2434175 - 21%
https://ibb.co/TLc98LZ

Dante's Inferno (USA) (v1.01)
Original 1770240
CSO 1468477 - 17%
CHD 1379283 - 22%
GZ 1424640 - 20%
PBP 5637395 increase 118%
https://ibb.co/w4y1qM9

Summary Saving
https://ibb.co/8x71gGW

@i30817
Copy link

i30817 commented Apr 28, 2020

CHD has support for parent-child relationship, which are both a interesting way to softpatch and a interesting way to save space in multiple cd games (by making cd 1 the 'parent' of cd 2).

I definitely don't recommend doing the second ofc, since it's kind of a 'semantic' mess - and i'm not sure you can do both at the same time. Or at least you can't do it twice (parent-parent2-finalchild).
It works like this: the child has the sha1sum hash of the 'parent' and when loaded you should provide both to get the 'complete overlay'.

Notice the complete lack of filename or paths here. It's to the application to use a convention to find the 'sets'. Either dats to find the filename, path conventions or a scanning step to collect the sha1sum of all chds on a 'game directory' are all options (i prefer the last option because it doesn't depend on users).

Supporting this is currently not supported in libchd (the non mame library handling chd for other projects).

@lamvuong2019
Copy link

The method 2 you describe is a form of deduplication, I think it will be a total nightmare and a massive performance impact if you don't have the necessary cache or cpu to data crunch.

I think the compatibility and easy maintenance of the chd is definitely a worthy method of backup.

@Sanaki
Copy link

Sanaki commented Apr 28, 2020

As the person who opened that issue, I agree clone support is wonderful, but it's also not terribly important for PSP. It would be quite nice for minor patching, sure, but it's really not as useful as for systems like the PS1 with tons of multidisc games.

Also, PBP is only used optimally for PSP, either for uncompressed PSN titles or for PSX-PSP (which is only of value on real hardware). Most of us ripped our PSN versions to ISO for ease of use though.

EDIT: Clones don't have any performance impact. CHD splits the image into tens of thousands of "compressed hunks of data" (hence the name), each of which is compressed individually based on which compression method is most efficient. Duplicated hunks are already referencing the same data, clones just reference almost all of the data from the parent, barring the few hunks that have changed.

As an example, this is the hunk type breakdown from a clone I made of a translation for a PCE-CD game:

     Hunks  Percent  Name
----------  -------  ------------------------------------
       424     1.5%  Copy from self                          
    18,319    66.4%  Copy from parent                        
     1,917     6.9%  CD LZMA                                 
       184     0.7%  CD Deflate                              
     6,743    24.4%  CD FLAC

To be clear, clones are a fantastic feature and were it supported I'd definitely use it, but I feel like worrying about them before they're supported by libchdr would be a bit premature. For now, let's just see about getting the basic format handled.

@i30817
Copy link

i30817 commented Apr 28, 2020

The method 2 you describe is a form of deduplication, I think it will be a total nightmare and a massive performance impact if you don't have the necessary cache or cpu to data crunch.

I would expect that chd is a better format than most for this and doesn't use silly half baked features like putting the original file in memory to 'patch it' which is the achilles flaw of every 'softpatching format expanded to cds' that causes all of them to be massive failures with cds - i'm looking at you BPS.

Libchdr may still screw up of course, but the format was done for this kind of 'deduplication' usecase.

And it's not like the native filesystem dedup works for isos or cue/bin anyway, the FS dedup is oriented to file - it does absolutely nothing to two 99% similar files, and compression dedup often screws up and turns that 99% similar into 20% similar if there are some insignificant but regular offsets discrepancies between two files. In fact, often a file compressor is so worried about compressing inside the file that it loses the opportunity to recognize that two large files are very similar since it forgets about the first file because of the compression window.

While chd knows those two files are related and uses filesystem block matching. Of course, even CHD may not help if those two cds are themselves using filesystem compression or cryptography inside their filesystem. Modern games are lame like that, use it case by case.

@hrydgard
Copy link
Owner

hrydgard commented Apr 28, 2020

OK, so a win, but not an enormous one. Some people might find it worthwhile and I do understand that it's nice with a common tool.

So if anyone is interested in implementing this, what you need to do is:

  • go into Core/FileSystems/BlockDevices.cpp/h, add a new block device class for CHD
  • hook it up in constructBlockDevice
  • fill it out with calls into libchdr
  • set up cmake and visual studio projects to build and include libchd
  • (minor tweaks like showing .chd in the file open dialogs, and stuff like that)

I probably won't get to this in the near future myself, busy with finishing up stuff for 1.10 in the time I have.

@hrydgard hrydgard added this to the Future milestone Apr 28, 2020
@Sanaki
Copy link

Sanaki commented Apr 28, 2020

I don't have the time or expertise to do it myself, but I did toss a $15 bounty on it in case anyone else does. Though bountysource seems to be having some issues right now, so it may take a bit to show up correctly.
https://www.bountysource.com/issues/52791233-request-support-for-chd-format

@lamvuong2019
Copy link

lamvuong2019 commented Apr 28, 2020

I am also willing to support the bounty, its also failing for me too. I tried to top it up with 15 also... getting a massive red errror something went wrong, I will try again later to see if it works.

It had an error but it went through anway, kinda strange anyway my pledge went through and the strangest thing is it is still staying at 0 dolar in the bounty... Hopefully someone pick this up as it is a big bonus for everyone.

Here is my proof for the pledge and no expiration.
https://ibb.co/qkKKqD5

@Sanaki
Copy link

Sanaki commented Apr 28, 2020

If it hasn't shown up by tomorrow I'll contact them about it. Given that the transactions are being recorded, it should catch up eventually.
PPSSPP CHD bounty

@LunaMoo
Copy link
Collaborator

LunaMoo commented Apr 28, 2020

Isn't PBP used for psx iso only?

I don't know if this is a good example or not

Final Fantasy Type-0 (English Patched v2)
Original 3084866
CSO Failed
CHD 2410963 - 22%
GZ 2434175 - 21%
https://ibb.co/TLc98LZ

Dante's Inferno (USA) (v1.01)
Original 1770240
CSO 1468477 - 17%
CHD 1379283 - 22%
GZ 1424640 - 20%
PBP 5637395 increase 118%
https://ibb.co/w4y1qM9

Summary Saving
https://ibb.co/8x71gGW

No, PBP is Sony's native container, it's used for all PSP games available from PS Store, it also can be used to compress iso's althrough there's only one software for that and it's win-only ~ https://sites.google.com/site/theleecherman/IsoPbpConverter

You probably used PSOne software to end up with larger file, it has no sense as PBP has better compression than CSO with large chunks and as far as I recall testing size was comparable to chd.

Adding to your results of FFT0, converted to megabytes for readability:

original(megabytes)                                   3 012
CSO1 lvl 9                                            2 497
cso2 with 4096 chunk                                  2 451
PBP                                                   2 404
cso2 with --block=32768 --format=cso2 --use-zopfli*   2 388
CHD                                                   2 354

So you save 50 mb on 3000 mb file(PBP - highest compression format supported currently in PPSSPP vs the unsupported CHD), that's roughtly 16 mb per 1000 mb's. Also the tool you use for CSO sucks, most modern CSO tools can support filesize above 2gb, I'd recommend [Unknown's] Max CSO, it also allows compression with larger chunks which is producing smaller file size than just using lvl 9 compression CSO.

Edit: included cso2 with [Unknown's] settings. So that's an 11,33mb difference per 1004mb between what we already have and CHD.

  • using --only-zopfli instead of --use-zopfli generated a file which was larger by ~ 150kb, but it ended much faster as it skipped other methods, so that might be worth considering.

In other words CHD compresses by 1.13% better vs currently supported format with highest compression.

Not that impressive as comparing it to standard CSO.

@LunaMoo
Copy link
Collaborator

LunaMoo commented Aug 18, 2022

Nice failed attempt to correct the facts.

*I am using MaxCSO. And inorder to do that max compression involves enabling something he author even disabled by default because it takes forever.

*As far as the hash, you got me there, I couldn't remember how to phrase it right when I wrote it.

*And I know that the emulators have to support CHD and not just RetroArch or any of that. That was a claim I never made.

As far as it being used primarily for Arcades, you haven't been keeping up then as it is slowly becoming the more standard way of storing PS1, PS2, Sega CD, and a bunch of other systems as well with PSP and Nintendo emulators being the main hold outs.

Saying that CHD is used mainly for illegal stuff is about accurate as applying it to CSO and RVZ at this point and debate-ably to 7-zip.

By comparison, CSO and PBP are used on PSP which is discontinued at this point which makes that less relevant by the year along with the fact that if I am playing it on an emulator, then I am not playing it on the console and vise versa which makes that comment even less relevant as this is about an emulator and not the console itself.

And, I never said that CSO was lossy, if it was, I would have stayed with ISO instead of CSOMax, never said it wasn't feature complete as, quite literally, all it is is a compressed image format. Not much you need to say beyond that.

And your last statement holds no water in regard to this thread as that only applies if the other formats offers functionality that the others lack that is of use to their users. And being able to directly place it on a PSP without converting it isn't a huge plus for most people who aren't transferring their games between emulator and console and supporting CHD doesn't mean removing support for CSO. So the fringe minority who do want to transfer their games (And assuming save files) back and forth between console and emulator can still do so.

Edit: And just out of curiosity, I tried using --use-zopfli to see how CSOMax was on max compression. The ram usage wasn't bad but the compression speed on my 4ghz CPU was glacially slow and you can see why it's disabled, it averaged close to 0.5MB/s. Glad I chose a smaller title to test with, the default I was getting 5-6MB/s average. When sites said it was on average 80 times slower for the gains, they weren't kidding.

Overall size on the file (Fat Princess - Fist Full of Cake - USA) Uncompressed ISO: 303,904kb CSOMax default: 195,552kb CSOMax MAX: 195,461kb (Less than 1 MB for this title) CHD (createcd): 190,787kb (4MB smaller than CSOMax and took seconds to do)

And from what I have researched, AT WORST CHD was on par with CSO while it averaged actually smaller with lower resource usage to do which my experiment panned out.

So, like I said, if you are just using the PC to hold your PSP games as you transfer them back and forth off your PSP and need the native support, CSO is your best option and being able to use them on the emulator is just a bonus at that point.

If you are using the PC to PLAY your PSP games it loses to CHD as the CHD at a minimum matches or beats it on compression with lower resources and quicker compression and, as you said yourself, it can store the hash value which makes it easier for groups like ReDump to catalog without having to waste more space and resources needlessly to do it.

So, outside of PSP compatibility, CSO either ties or loses to CHD across the board. I mean at least RVZ has a purpose that CHD doesn't fill given their ability to handle the junk files while still being lossless.

I hate checking this spammy issue, anyway:

  • MAXCSO "takes ages" to compress only because by default it tries to compress each block with various compression algorithms then keep the one that worked best, this is it primary feature and what actually allows for lower file sizes with some games despite CHD generally using stronger comprssion(since it's NOT universally stronger for every data type), you can limit it to algorithm of your choice and also specify number of threads to use and block size, depending on your options it can be really fast, but then you're loosing on the primary feature.

  • It doesn't have hidden options, in fact it's opposite to CHD, when you run MAXCSO without any argument you get an actual list of arguments and options, doesn't need to search through some archaic places over the net nor look at mame's codebase to figure it out as it's the case with CHD which sucks hard the user friendliness of CHD is under the sea level compared to MAXCSO despite both being CLI.

  • MAME is the king of piracy in emulation world, many of us don't want to be linked to that side of emulation, CHD as of now is unfortunately part of it, also being part of it, it's developed around MAME specific needs which are completely different from what modern emulation needs. I'd rather support extracting 7z to RAM since on modern hardware many of us cache ISO into RAM anyway and if you cared soo much about size that would win.

  • If you had 0,5mb on a 4ghz cpu then what can I say, time to move on from your P4:), it's slow, but not that slow and again that's MAXCSO feature, it has worse compression algorithms, but it compresses with multiply algorithms and then picks the best one, which is why it can achieve better results in some cases and in others it's really close. Saying you'd need better compression when the difference is around 1% is like really bending reality to your needs as even CHDMAN fans want a different algorithm with slightly worse compression, but better decompression for use on mobiles:).

Anyway I was never against the chd format itself, but against:

  • it coming from mame and being full of stuff we don't need nor care to deal with,
  • saying it has much better compression - which is just bs, the difference is like 1% which is not "much" by any means and since maxcso compresses the same data a few times then keeping whichever compression was the best which CHD does NOT support, I did had case where MAXCSO had better compression again due to MAXCSO feature of using multiply compresion algorithms which CHD lacks(I think it was some MH game, but it's somewhere above in this spammy issue),
  • saying it has essential emulation based features - it does not as explained with hashes last time it doesn't benefit PPSSPP at all since we have param.sfo files provided by the games itself and don't need to store hash which is also not usefull for validation, it can tell you for example "you downloaded a copy made of proper dump", it CAN'T tell you that your copy is STILL a proper dump and file corruption is common within PPSSPP(mobiles) userbase,
  • same can be said about softpatching, which in case of PSP games would be better on an emulator level, not on file format level, since for us it would only be useful for something like fan translations and then those always keep compatibility with original hardware, we don't have much use for it otherwise, even for game updates(which btw require use of actual hardware) it's better to develop a sideloading of decrypted files since due to how updates on PSP work, game could still technically access both, updated executable and old one, softpatching doesn't solve that,
  • also I'm very much against any arguments surrounding it around "libretro depends on it, you provide it" that project it leeching on emulation for long enough and still their port of PPSSPP sucks, they should start working instead of depending on other project's work.

@i30817
Copy link

i30817 commented Aug 18, 2022

Respectfully, who is talking about libretro on the issue? The whole reason of 'why chd' has nothing to do with libretro. It has everything to do - to me - with delta chds, which are the only softpatch format for isos around.

If you think you're going to get translations 'at the emulator level' you're completely going against portability - i'd prefer not to have emulator lock in, yes - also, try to convince romhacking.net to convert all of its psp patches.

I don't have the energy to go against the defaming about 'this is just for piracy' going on in the rest of the rant. Is 'cso' just for piracy? Why not, because it can be used on the psp? Doesn't it make it more likely to be used for piracy?

This is just weird format hate. There are reasons not to use chd - you want a psp compatible compression, or you want to run compressed isos on platforms that can't open more than one file per user process at a time, or you want to rename dumps against redump or redump updates (for now), but this paranoia about 'associated to MAME' or 'libretro wants this' is just bizarre. Remove or sabotage the upstreamed libretro backend if you hate their guts that much and quit transferring that hate to a fileformat from another project.

@ileathan
Copy link

ileathan commented Sep 10, 2022

Doesn't the closed PR here add this feature?

I think you are right. Did you try it out though? From what I read it looks like someone made that update but it was not supported by the repo developers? So I'm scared to switch over to it only to have no updates. But at this point I just really want to use CHD for many reasons.

@0x90shell
Copy link

Doesn't the closed PR here add this feature?

I think you are right. Did you try it out though? From what I read it looks like someone made that update but it was not supported by the repo developers? So I'm scared to switch over to it only to have no updates. But at this point I just really want to use CHD for many reasons.

I didn't compile the changes, but was just reading the related discussion where it seemed to die on the vine for no discernible reason. Same dude did the PR to support CHD to PCSX2 iirc.

@crashGG
Copy link

crashGG commented Sep 20, 2022

Tested MaxCSO vs CHDMan on 3 random games.

CHD was both smaller while taking a fraction of the time to compress in all 3 cases on default setting. I know that MaxCSO had a higher level of compression flag, I tested it once, it was a minor improvement in size compared to default settings but took ages to do and CHD was still smaller.

Many people who say that the compression ratio of chd exceeds cso ignores the fact that chd uses logical blocks of 19584 bytes for compression by default, which is 8 sectors in size (chdman uses full 2448 bytes as logical sector). Then under the same conditions, it is fair to compress into cso by setting the block size to 16384 bytes. If you use maxcso compression, you need to use the parameter maxcso --block=16384

@0x90shell
Copy link

Maxcso takes ages to compress at max whereas chdman is much faster by default with better compression. Outside of the initial time to compress, CHD files are also supported by Retro Achievements.

@adamsbuilds
Copy link

CHD for performance. would love to see it implemented

@TomTurbine
Copy link

TomTurbine commented Dec 9, 2022

Responding to LunaMoo

MaxCSO takes a while while still being bigger than CHDMan by default and resulting in bigger files. The main feature that they tout to improve compression even further takes ages longer and still doesn't match CHD's default. There might be some theoretical settings that give it better compression than CHD but at what compression speed? It's not a lie, but its been tested repeatedly, at the defaults, CHD beats CSO in speed and compression ratio and while using it's flag to get even better compression that STILL doesn't match CHD. Maybe if MaxCSO changed to new defaults to at least match it's speed or ratio then that argument would be moot but its not.

And I understand you dislike Mame because you consider it piracy, but that kinda is beyond pointless in this one because pretty much every emulator can be used for piracy, including this one, and refusing to use an open source item that works out to be better overall just because of that is, to use an old saying, just cutting your nose off to spite your face.

And your support for 7-zip does have some major drawbacks compared to CHD. Having tried that on another system with equally big files, that leads to pretty long load times as it has to actually decompress the entire thing into ram which can add upwards of another 30 seconds to the load time if you are using a Hard drive over an SSD. While the CHD can match a 7-z with ultra settings and a 192MB dictionary. Adopting the 7-zip wouldn't be a win over CHD, it would be a sad consolation prize that gives massive drawbacks over actually supporting CHD. One of the actual advantages of CHD is that it can run the game while compressed unlike the 7z that requires decompression first to run and PSP titles aren't like SNES where it is so small it takes a fraction of a second to deal with.

As far as the 0.5MB bit, that was with the MAX settings selected on an 8 core processor, the default settings which much faster but still slower than CHD's defaults. Updated since to a Ryzen 9 5900x and haven't tried it recently but honestly don't expect a miracle. The old processor wasn't exactly giving me a problem it anything either and the upgrade was more about scratching an itch and giving the older one to a family member but having an upgrade go from running a game at 160FPS to 200+FPS doesn't make much of a difference to the end user. Borderlands 2 only needs to go but so far and no CPU bound programs I used ever gave issues except CSO compression.

Sir, the fact of the matter is CSO is an antiquated format at this point given the more supported options. BY DEFAULT, it has been beaten by CHD in both speed and size, if MaxCSO wants to change their defaults in a newer version to at least match the compression ratio of CHD without taking 10 times longer than it or at least beat it for the added time, you can have a point.

But at this time, there is little point to having CSO over CHD. CSO's main selling point is that the PSP can actually read it which doesn't really matter when the PSP is a dead system that is far gone that it is hard to get a hold nowadays so your choice of playing it either through emulation or not at all for most people.

Now, if CSO was something like RVZ is for Dolphin where it actually had a real benefit over the other formats, I could see that. But at this point CSO over CHD is just being stubborn to be stubborn with no benefit to the emulator or its user base. There is a reason that virtually every other disc based emulator outside of Nintendo (RVZ) and Xbox (XISO) has adopted CHD support and not CSO.

Heck, even at this point, I have recommended that XEmu emulator add support for CHD compressed XISO as it even offers major benefits there. I know it feels like people are trying to use it for everything, but that is because, for disc based emulators, it beats other disc image formats where it counts.

Refusing to use CHD because it came from another emulator who open sourced it for anyone to use is foolish if it improves your own software in the process. And not including that software doesn't fight piracy in any way, shape or form, it just lowers the usability of your own software in the process by omitting that one feature that many want and has tangible benefits to the end user.

And to say its because Mame is associated with piracy is pointless as emulation as a whole is associated with it outside of the community itself. My friend actually used to collect arcade machines before he got married and we even got gold memberships to a local barcade for the systems he donated to them but the fact of the matter is I would still prefer mame to the cabinets he had because I have never been good on a joystick and preferred a directional pad for most things with the only exception on those old things being Daytona USA because the steering wheel setup can't be matched on a typical emulator setup. Also, never try and outrace a man who owns the machine, he has every turn on every track memorized into muscle memory, same goes for Mortal Kombat.

If the author refuses to support CHD, that is on him, but to claim CSO is the better option is untrue for the average end user. And to claim it is "good enough" still really doesn't hold much water for many users as they only use it begrudgingly because its the only real format they are able to use.

The instant CHD support was enabled, you would likely see new users wanting CSO drop like a rock with a large portion of older users actually transitioning over as well. PPSSPP is really the only thing I know of giving CSO any relevance at all nowadays given the overall death of disc based games and systems outside of consoles.

@adamsbuilds
Copy link

adamsbuilds commented Dec 9, 2022 via email

@hrydgard
Copy link
Owner

hrydgard commented Dec 9, 2022

Nobody is refusing to support CHD, it's just not a priority.

Again, if someone contributes fully working CHD support and commits to help update it as needed, it'll get in. As for me, I prefer to work on things that can actually improve gameplay and graphics. ISO formats are not very interesting to me and CSO works fine right now.

@nickkeane
Copy link

@hrydgard would SleepyMan be willing to maintain it? That's the person who created the original pull (why is it closed?) #13810

@TomTurbine
Copy link

@hrydgard would SleepyMan be willing to maintain it? That's the person who created the original pull (why is it closed?) #13810

I don't think so, I just checked his profile and I don't see any activity on it for a long time.

@TomTurbine
Copy link

@hrydgard

Understood, I was not directing my responses towards you. I understand your viewpoint and have no issues with it as it is a completely acceptable answer that it isn't a priority given all the other work.

My comments was directed more a LunaMoo who seems to have this irrational hate boner for CHD and Mame almost like he designed CSO himself and can't fathom why anyone would want something better.

CHD really seems like the better way to go in the long term but it isn't a short term priority. Hope to see it included eventually but it's understandable why it isn't yet given all the other work that can be done.

@Double-0-seven7
Copy link

Double-0-seven7 commented Aug 12, 2023

I don't have the time or expertise to do it myself, but I did toss a $15 bounty on it in case anyone else does. Though bountysource seems to be having some issues right now, so it may take a bit to show up correctly. https://www.bountysource.com/issues/52791233-request-support-for-chd-format

Damn how this gets a bounty while the state of ad-hoc/multiplayer support and other preservation related functions on the emulator are way worse since the emulator inception.
And you even have an alternative compressed formats (actually 2 other) that you can use.
for now.

@ghost
Copy link

ghost commented Aug 30, 2023

@hrydgard
Copy link
Owner

I don't want a bounty, but I gave it a shot and came up with something much smaller and maintainable than the previous attempt: #18198

Note that I still recommend CSO.

@hrydgard
Copy link
Owner

Basic CHD support (without parent/child iso support, since it's unclear how to do that efficiently without scanning every file in a directory) is now in.

@RinMaru
Copy link

RinMaru commented Sep 29, 2023

Basic CHD support (without parent/child iso support, since it's unclear how to do that efficiently without scanning every file in a directory) is now in.

Ive been looking everywhere but what is Parent/child iso support? Multi disc games into one CHD?

@Danixu
Copy link
Author

Danixu commented Sep 29, 2023

Basic CHD support (without parent/child iso support, since it's unclear how to do that efficiently without scanning every file in a directory) is now in.

Ive been looking everywhere but what is Parent/child iso support? Multi disc games into one CHD?

Parent child relationship is like in the real life, means a game that is "derivated" from another. For example a translated game, hacks, and if I'm not wrong different regions. It's a way to keep related Roms together.

@RinMaru
Copy link

RinMaru commented Sep 29, 2023

Basic CHD support (without parent/child iso support, since it's unclear how to do that efficiently without scanning every file in a directory) is now in.

Ive been looking everywhere but what is Parent/child iso support? Multi disc games into one CHD?

Parent child relationship is like in the real life, means a game that is "derivated" from another. For example a translated game, hacks, and if I'm not wrong different regions. It's a way to keep related Roms together.

So like final fantasy 0 was a two UMD game it can be stored as one chd? Thought CHD didnt have that yet. I remember ppl wanting CHD to store multi disc games as one file like sonys PBP format.

@Tuxie
Copy link

Tuxie commented Sep 29, 2023 via email

@Danixu
Copy link
Author

Danixu commented Sep 29, 2023

Basic CHD support (without parent/child iso support, since it's unclear how to do that efficiently without scanning every file in a directory) is now in. Ive been looking everywhere but what is Parent/child iso support? Multi disc games into one CHD? Parent child relationship is like in the real life, means a game that is "derivated" from another. For example a translated game, hacks, and if I'm not wrong different regions. It's a way to keep related Roms together. So like final fantasy 0 was a two UMD game it can be stored as one chd? Thought CHD didnt have that yet. I remember ppl wanting CHD to store multi disc games as one file like sonys PBP format.
No. If a game had for example one release for USA and one for Europe, one release (the parent) would have a normal full size CHD and the other release (the child) would have a separate tiny CHD containing only the difference from the other one, and you'd need both files to play it.

I forgot this point, the child are "differential" copies.

Multi disk is not supported by the CHD format itself as long as I know, so there is no way to do it.

@RinMaru
Copy link

RinMaru commented Sep 29, 2023

Basic CHD support (without parent/child iso support, since it's unclear
how to do that efficiently without scanning every file in a directory) is
now in.

Ive been looking everywhere but what is Parent/child iso support? Multi
disc games into one CHD?

Parent child relationship is like in the real life, means a game that is
"derivated" from another. For example a translated game, hacks, and if I'm
not wrong different regions. It's a way to keep related Roms together.

So like final fantasy 0 was a two UMD game it can be stored as one chd?
Thought CHD didnt have that yet. I remember ppl wanting CHD to store multi
disc games as one file like sonys PBP format.

No. If a game had for example one release for USA and one for Europe, one
release (the parent) would have a normal full size CHD and the other
release (the child) would have a separate tiny CHD containing only the
difference from the other one, and you'd need both files to play it.

Oh that makes sense. Thank you for the explanation

@BParks21
Copy link

BParks21 commented Sep 29, 2023

So what is an example of a game with this parent child relationship? Something like Birth By Sleep Final Mix with an English patch/translation cannot be used as chd?

@Danixu
Copy link
Author

Danixu commented Sep 30, 2023

So what is an example of a game with this parent child relationship? Something like Birth By Sleep Final Mix with an English patch/translation cannot be used as chd?

For normal people this feature is not important. Only is important if you want to preserve the PSP library with all games and all versions saving some space. Anyway by the nature of the differential patching, this will not work in almost all PSP games (if not all), because normally the games are ISO rebuilds and they don't share data positions. This feature is more important in other consoles ROMS or games patches using ppf patchers or similar.

Anyway, maybe an example can be to have the Final Fantasy Type-0 game which is a JP game (which will be the parent), and the English patched version (child).
If the patch is differential and not an ISO rebuild, the resultant child can be much smaller and you will have both games versions preserving the original one.

Best regards

Repository owner locked and limited conversation to collaborators Oct 5, 2023
@LunaMoo
Copy link
Collaborator

LunaMoo commented Oct 5, 2023

(...)For normal people this feature is not important. (...)

  • just as the whole chd container in PSP emulation.

(...) Final Fantasy Type-0 (...) If the patch is differential and not an ISO rebuild, the resultant child can be much smaller and you will have both games versions preserving the original one.

  • bad example, FFT0 english patch not only rebuilds two iso's into one, it also has to do what most PSP game patches are doing that is rebuild archives on the game disc which tend to be encrypted and compressed, due to that differential patches for PSP games tend to take 99% of the original game size,
  • the exception is tiny eboot patches, but those can be loaded as cheats which is better for anyone that doesn't want to permanently patch their game anyway or are working on something and doesn't want to patch it every time to test stuff.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.