Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Add support for loading CSO-compressed ISOs #504
This adds support for loading files compressed using CSO, a format that is often used for PSP ISOs. It just compresses blocks of the file, so it can work with many different file types.
I've written a CSO compressor which supports >4GB CSO files. Not all existing CSO tools support > 4GB files unfortunately, but some others do.
This patch also works fine with such ISOs, e.g. Wild Arms 1: Alter Code F.
This implementation is similar to PPSSPP's, although for simplicity it lacks a few optimizations. If there's interest, I can probably implement async reading and the multi-block read optimization. But, I think this can/should be merged beforehand, as it already works fairly well.
This isn't intended to replace or be better than the existing gz feature. But supporting more file types is not a bad thing, and there are more tools available which support random-access in CSO files (for hacking or etc.)
Actually, that's not always the case. In the only size comparison I did, the size of a 7-zip .gz (max compression) file + the pcsx2 index file was 1.8% larger than the cso file (using
Again, that's not to say cso files are better than gz files, but I don't think it's fair to say that gz files are definitely better than cso files either.
Well, both of them use zlib, so it'd be a fair assumption that their compression ratios would be similar (and probably configurable within the same parameters and performance ranges), up to the facilities each use to speed up seek (blocks and indexed with the gzip implementation, blocks with the CSO decompression).
Which open source tools can handle this format (compress/decompress/modify/etc)?
@unknownbrackets, the patch looks reasonable, though obviously I'm not able to tell if it actually works or not after just reading the code, so:
What do you think?
There's a listing of tools here that can handle the format:
Of course, maxcso itself (as previously mentioned) is the tool I'd recommend (I may be biased.) It supports using multiple threads, uses multiple compression method trials based on options, and can generate the CRC of a CSO file's uncompressed data.
Booster's original version is arguably the original "specification", but I've created a document here:
I'll add a link to that in the code.
2 - They don't really. Unless you want
I've tested this patch in pcsx2 with several ISOs mostly of different sizes (e.g. ~6+GB, 1GB, etc.) It works with non-English filenames, but it uses the same methods of file access as the gz compression type, so it will have whatever known bugs that has.
Performance: I have not carefully measured performance. This is a newly supported format and I've primarily ensured it does not impact existing performance. However, based on performance tests (and improvements, some of which incorporated here) made with PPSSPP, I'm fairly sure it won't perform poorly. That is, this is already based on an optimized implementation.
It could (as already mentioned in the description of this pull):
The first is not even implemented with gz file support. The second did not give a very big improvement on PPSSPP, but did provide a measurable one, and so as I mentioned, I can implement it after this is merged.
Also, I suppose it could:
Compliance: The only feature of the format that this patch lacks is alternative versions:
Neither have seen any significant adoption. This patch will reject files of either type as invalid.
4 - Caching: I can test that. I remember trying the CachingFileLoader we have in PPSSPP and it not improving things much, but maybe the semantics of PSP game access are significantly different (PPSSPP's caching includes read-ahead and is tuned to play video/etc smoothly when reading over the network, though, so its semantics are surely different from the cache here.)
This implementation does use a small cache, which is the last-decompressed frame (important when games read small numbers of bytes sequentially.) At least in PPSSPP, I found that this was by far the most common case of a block being read multiple times. It's definitely good for performance, although deflate is fast even without it, of course.
5 - Well, I noticed that CompressedFileReader itself was in a completely different file:
So I assumed this was intentional. If that's not the case, sure, I'd be more than happy to split them out. I also would prefer the two classes be in separate files with their own names. Is CDVD/ fine?
YACC just uses CisoPlus internally (it's just a frontend.) I don't know if it has large-file support, but at least for < 2GB ISOs I'm fairly sure it'll work.
Otherwise, I would recommend the following to answer your question (since CisoPlus is not open source):
Note that PSP UMDs could only be 1.8GB. It won't be shocking if it does not support larger files (since YACC and CisoPlus are largely PSP-centric tools.)
added a commit
this pull request
Apr 15, 2015
So, I tried using the chunk cache thing.
Here's what I found:
What this means to me is that the overhead may not be worth it, considering how fast deflate is.
So, that led me to do the read from zlib for cache hits or misses, and time that vs. the overhead of cache lookup + write on miss.
Here's a case where I let some music loop to get a ~25% cache hit:
In this case, 1.09 represents the total time it would take to read without the cache. 0.656 is the extra time taken by using caching.
It proves out with other tests, but
It would probably help more if you are in an area and music is looping a lot, the game is re-reading from the disc when you enter battles, etc. So, it can most likely win. However, it doesn't look like the win will be huge, and it will most likely increase load times to get there.
Therefore, I don't think it makes sense to add ChunksCache to the CSO implementation.
Edit: oops, I was actually counting cache lookups in the read time as well by mistake, so it's worse than that (although probably not much, I think the malloc/memcpy on write is slower.)