[enhancement] Zip support for multi-disk for computer emulators (mostly lr-vice and lr-puae)

[valerino]

Description

As far as i see, the current .m3u and Disk Control support relies on the fact that:

• all the disks are in the same .zip (so an .m3u is automatically created and disks are selected through disk control)
• OR disks are already uncompressed, so there's no problem at all.

Problem happens, since most people has their collections stored as 1-zip per disk, i.e. when you want to use a game with 3 disks: for Amiga you would have 3 zips, for c64 you would have 6 zips (3 zips for side A + 3 for side B), and so on.

In this common settings you can't load any further disks from the disk control interface since, citing from the docs (taken from lr-vice, for example):

Note : zip support is provided by RetroArch and is done before passing the game to the core. So, when using a m3u file, the specified disk image must be uncompressed

Of course this could be handled by recompressing all the collections in 1-zip-per-game, or for Amiga recompressing all the zips to .ADZ (as lr-puae docs states this is supported natively by the emulator), or keeping the collections uncompressed.

But, i think it's trivial to implement this correctly in Retroarch itself, since it generically affects many emulators.

Expected behavior

start the emulator with loading the disk 1, then (through the disk control menu) load disk 2 when needed and the game continues correctly.

Actual behavior

after inserting disk 2 and resuming emulation, it won't load since it doesn't get uncompressed.

Steps to reproduce the bug

I describe the steps to reproduce here just for Amiga (lr-puae), but it's general for all affected emulators.

1. get a multi-disk game for Amiga (i.e. Fire & Ice), with both disk 1 and 2 as separate zips.
2. run disk 1 selecting it as normally using 'load content', get to the point where it asks for the 2nd disk (right after the title screen)
3. quick menu -> disk control -> load new disk -> select disk 2
4. make sure it is inserted, resume emulation

also using an .m3u pointing to the 2 zips still doesn't work (same behaviour).

it only works (both with manually adding the disks or with the m3u) with uncompressed disks.

Bisect Results

i think it was implemented this way ..... sincerely, i wonder why this simple scenario has not come to mind ....

Version/Commit

• RetroArch: nightly mar 15/2020, git version: 7c57124

Environment information

• OS: Android (but i think it doesn't matter here...)
• Compiler: Clang/LLVM 8.0.7

[i30817]

libretro-uae supports zip already (and lha and whdloads and...). The only thing missing for now is chd.

If you meant 'disk images inside a zip', i'm not sure. I'm speaking of the (far more valuable imo) 'zip as a amiga filesystem'.

(Which doesn't work for all games, since the amiga is a foreign filesystem with its own rules, but works for many and most whdloads).

I prefer the retroplay whdload collection over TOSEC even if it's incomplete and buggy sometimes. Not having to care about disk changes or protections or incompatible st palettes is fantastic.

If you're talking about the m3u support only, jdgleaver is thinking of creating a PR to create them automatically, and i have a script that does the same: https://gist.github.com/i30817/7dccadc56bfa63f344032c957d8d6cf9 first argument the file tree to be scanned second argument the dir to place the m3u in (iirc). You can edit that to add adf or whatever you need by editing in the last adf

if [ "${#names[@]}" -eq 0 ]; then
	names=(*.{mds,cdi,img,iso,adf})
fi

and by changing the code that matches 'cd' or 'disc' to also match 'disk # of #' for tosec which is more complicated but possible.

[valerino]

the point is lr-puae (and lr-vice, and all other cores) supports zip, sure, but for single disk games only. you can't use multi-disk zips easily without decompressing them first.

as an example, take an Amiga 2 disk game and make a .m3u for it: in popular collections (TOSEC and others) each disk is separated, so you have 2 zips (each one containing an .ADF).

so you will end with this .m3u:
(game.m3u)
disk1.zip
disk2.zip

but, in the current implementation it doesn't work, since retroarch doesn't unzip the files first (and then pass them to the core UNZIPPPED).

so, as now, you need to have unzipped disks for the .m3u to work:
(game.m3u)
disk1.adf
disk2.adf

this works perfectly, but again one needs to have everything unzipped (which is not common).

i think it's trivial to support this in retroarch directly (just unarchive ALL the compressed disks in an m3u THEN pass them to the core i.e. from a temporary folder), so my proposal.

also, this will fix automatically all cores needing multi-disk (lr-vice has the same problem, and other cores probably as well) without touching each one.

i made some scripts too on my github, to deal with this.... but having this supported in retroarch directly would be great!

[i30817]

i think it's trivial to support this in retroarch directly (just unarchive ALL the compressed disks in an m3u THEN pass them to the core i.e. from a temporary folder), so my proposal.

Ok, so you basically want to create a 'retroarch' rule that all zip files inside a m3u are supposed to be single roms instead of m3u. This is kind of sane because i don't know of any cases where you want a m3u and multiple zips where that isn't the case, but notice that a tool that autogenerates m3u files can easily create them for single 'roms' too (for consistency if nothing else), and that would mean that there is the danger of RA trying to unzip something that the core wants to handle itself (things like prboom or libretro-uae hard disk zips).

This whole area of RA is so full of corner cases - doesn't work with softpatching, broken multi-session cds with seperate files, extract files to scan, huge disk stress etc - i gave up on using compression that is visible to retroarch (i use filesystem compression). Thankfully libretro-uae WHDLoads use .lha compression and retroarch has no clue how to butt in that.

[valerino]

Ok, so you basically want to create a 'retroarch' rule that all zip files inside a m3u are supposed to be single roms instead of m3u.

well, m3u is specifically meant for supporting multiple disks (or i miss something?). so of course, each .zip in an .m3u is a single disk.

This is kind of sane because i don't know of any cases where you want a m3u and multiple zips where that isn't the case, but notice that a tool that autogenerates m3u files can easily create them for single 'roms' too (for consistency if nothing else), and that would mean that there is the danger of RA trying to unzip something that the core wants to handle itself (things like prboom or libretro-uae hard disk zips).

shouldn't be a problem for cores. in 'my way' its a generic approach ,a core (no matter what) will always receive as input from reatroarch an uncompressed content. just, retroarch decompress everything before passing to the core.

i know there's other solution for this (filesystem compression is ok, but is only for ntfs, one may prefer to use exfat or other filesystems not supporting compression), and maybe this could be handled just with a retroarch configuration option ('decompress content first', or something like that) to be less invasive as possible.

[sonninnos]

I've been planning to finesse the ZIP and M3U support in PUAE some more after the latest automatic M3U creation feature. As in also handle them properly via the disk control insert, and not only when starting content. That would and should cover this issue too.

VICE needs a lot more work with ZIPs, as it still goes with the default RA route, which only extracts the first file.

[valerino]

great you're working on it at least for puae! but, wouldn't implementing this thing in RA directly fixes automatically all cores ? something like:

1. you select n disks from RA
2. they get decompressed in a temporary folder
3. an m3u is automatically generated with the decompressed content
4. core is started with the m3u.

this way there's no need to implement multiple disks decompression in the cores at all: RA handles the decompression, cores don't even notice ... they work as normal loading an m3u pointing to decompressed disks.

this will work for everything, except very particular situation (now that i think of :) ) where you may need to write to the disk (i.e. a savegame, or something). this of course may be handled (i.e. recompress the affected temporary disk), but its uncommon and probably of secondary importance.

[sonninnos]

In the meantime while the optimal method is brewing, here are some additions to the UAE core addressing most if not all of the cases: libretro/libretro-uae#264

Simply it allows:

• Making playlists of zipped files, as in no need to extract unless of course write access is wanted
• Inserting ZIPs (single and multiple disks) and M3Us properly via Disk Control

I'll try to add the same features to VICE too.

[valerino]

Inserting ZIPs (single and multiple disks) and M3Us properly via Disk Control

great work! the above is the basic feature for what i meant, everything else is welcome but you may think it as an extra :)

[RobLoach]

"Extract whole archives in some cases" could be related: #4331

[sonninnos]

And here are the same capabilities for VICE: libretro/vice-libretro#226

[valerino]

amazing! so this can be considered done!
unless keeping this open for other platforms.... ?

[Zephyr-Battiassi]

Could I suggest this feature to be applied to other cores like Caprice32 or CrocoDS (Amstrad)?

Forget this, I found this feature is already supported on Caprice32 core.
Thanks anyway.