RFE: Optionally automatically group cds/content files in playlist

[i30817]

First and foremost consider this:

• Only RetroArch bugs should be filed here. Not core bugs or game bugs
• This is not a forum or a help section, this is strictly developer oriented

Description

Refinement of #6324 (comment)

I was convinced by @Ferk that the VFS component or scanner changes of the previous idea were unnecessary so here is the much simpler version that instead changes playlists views a bit more substantially.

Expected behavior

1. There should exist 'cosmetic' setting to group files of the same game on a single playlist entry, even if they're actually multiple files (typically Disc 1, Disc 2 etc).
2. This 'facade' should use a heuristic. I've had 100% success rate on the ~= 10% of the redump psx USA collection I have using the heuristic present here.

The heuristic is simply to order the files by 'natural name' (not the ordering that screws up and places '1, 11, 2'), then iterate over the list and group files by finding common substrings that start from 0 pairwise, until you find a pair that doesn't end in a " [([]cd #*[)]]" or a " [(]]disc #*[)]]" (or where the number is different from previous +1) and when you do, finish the group (and create a new one on the next iteration). It's simple, especially since RA scanner already only picks up content files (though that might itself be a problem when it picks up both cue/iso and the iso that the cue refers to, but that's a different scanner bug that may or may not exist).

The heuristic works on the script even with different games on the same directory (creates different m3u for each game), so it will also work with a more abstract 'playlist'. For instance:

[Arc the Lad III (USA) (Disc 1).cue, Arc the Lad III (USA) (Disc 2).cue, Bushido Blade (USA).cue]
would turn into:
[ {Arc the Lad III (USA), [Arc the Lad III (USA) (Disc 1).cue, Arc the Lad III (USA) (Disc 2).cue]}, {Bushido Blade (USA), [Bushido Blade (USA).cue]} ]

(it's possible to support content in different dirs if the sort/group heuristic only operates on the filename part of the path, but of course the 'real' paths are complete unlike this simplified example).

This derived struct list could be used both for displaying only a single entry for game in playlists and for filling the cd-change entries automatically, allowing users to use the shortcuts without creating a manual m3u or adding each cd from the GUI.

The short name is the filename not including the match (not including the "(Disc number)" metadata), ie: not the parent dir, etc. If match found, remove extension, trim and remove any duplicate space and If the match is not found, don't include extension anyway.

existing m3u files on a playlist should be ignored in this grouping and appear 'as is' because they're already groups of files. Although a different issue could be hiding from the playlist files that exist already on the m3u if you want to remove duplicates (similar to the hypothetical cue/iso, iso problem above, so the place to fix this might be the scanner)...

Three complications:

1. I'd also request to change the core initialization code to make it possible to apply the 'short' naming as a per-game configuration hook (that works for all cds) when this mode is enabled (or maybe even always). Something like this is a major usecase to use m3u currently and it would be nice if a feature that makes m3u unnecessary in most cases offered the same function.
2. you still need to care about the loading savestates that refer to a different disc from the currently loaded one. The mednafen core copes with this somehow, so i'd expect that the same solution could be used on other cores, if it isn't solved already. This is very important for autoload savestates too, no point in starting to boot cd 1 if you're going to load a savestate from cd 2. If you accept suggestions, I think it would be valuable to to make these 'groups' savestates be part of a group by themselves and when loading a savestate first load the relevant cd/floppy if necessary. Converting existing individual cd/floppy savestates to this group could be done, but might be problematic because of the pigeonhole principle. Or it might not, considering that the savestate list is about 100 or so already. I'd suggest sorting by creation order a collection of all savestates and take the last 'min(total game savestates, nº max savestates)' if you convert, not just a concatenation.
3. not all types of playlists can do this grouping. Floppies are a possibility (depending on dumping groups naming); but a hard exception is the x68k playlist. There is really no point on doing this for it because it has two floppy drives, so the typical admissible group distribution for floppy change is more like a unknown subset of nºfloppies_Permutations_2, which isn't very helpful. Not sure if MAME cores need this too, because mame already knows what zipfiles names are required to load a game even with split sets (but i may be wrong, i haven't tried MAME emulation on systems with multiple cds or floppies). I'll mention that a alternative to using the heuristic that will work for x68k nPermut2 subset is to store the admissible subset on a libretro-database (a bitset seems feasible) and match files to those sets by CRC. However, that seems much more complicated (finding out which permutations the game needs which means playing the games from start to finish, creating a new db with this info, etc)

Actual behavior

Nothing, no such option.

Steps to reproduce the bug

1. [First step]
2. [Second step]
3. [and so on...]

Bisect Results

[Try to bisect and tell us when this started happening]

Version/Commit

You can find this information under Information/System Information

• RetroArch: [version/commit]

Environment information

• OS: [The operating system you're running]
• Compiler: [In case you are running local builds]

[i30817]

Hope this idea has a better response than the previous. Seems neater anyway.

[Papermanzero]

Grouping or a folder structure would be great.
Especially for other games apart from discs it would be helpful.
Examples are Nintendogs (4 DS Versions), Megaman Battle Network 3 (3 GBA Versions), Pokemon (2-4 GBA/DS/3DS Versions) etc.

[i30817]

This wouldn't work for those groups. It's not intended for that and there is not much point. If you want all the versions keep them on a separate zip or unzipped and choose the individual games on the playlist to play.

[Papermanzero]

But why not aiming for those two use cases?
The m3u does not care if you use Floppies, Carts or Discs.
Grouping has the benefit of clustering game content together which belong together.

[i30817]

Because this a heuristic. There isn't a good heuristic to group versions at the same time as discs and no reason to (unlike groups changing cds or floppies which is something that can help to load savestates).

Well there is 1 usecase (using the same config for all the versions of the game), but i'm not confident that a workable version heuristic can be worked and work at the same time with cd change. Instead i'd prefer those cases be handled by #6089

edit: edited out pointless digression

[Papermanzero]

The reason is "overview in the UI".
In general the use case is to cluster different storage medium so that the loader can use one of the version within the clustering.

The user in front of the loader has to select one of the storage mediums.
If he begins with medium A. Everything is fine.
But if the user wants to continue the game, he has to possibly select medium B.
Therefore the option to choose one medium is necessary.

The algorithm (I guess this is your heuristic) is always the same. Therefore it can also be used for game versions.

[i30817]

What's the point of sharing savestates or shortcuts for changing cds on two different versions of a game? For cd change it makes sense since it's the same game and the user will use all cds in a playthrough so loading different savestates inserting a cd makes sense. For different versions changing medium (cd) is just playing a new game so the savestates should be separate and the entries too.

No, i won't recommend grouping versions on the same entry.

And to be honest, i find goodroms 'single zip for all versions' a waste of hd space and bad design too, so if you're trying to support that usecase by trying to use this issue, well, you should make your case on another issue. I'd expect that even if it was supported somehow, it would end up with different entries for each rom.

This idea is not for zips (only) but for all the content files that RA scanner can admit into a playlist. I myself only have unzipped roms and use external filesystem compression.

[Papermanzero]

I understand your point now. But what you would like to have is core dependet. Retroarch as UI can only realize your proposal to a certain extent.

[i30817]

edit: I actually rather doubt this can cause widespread trouble even if it's not whitelisted or blacklisted. Maybe it can disable game configs setups for single games depending on how that is handled (but i'd still like the great benefit of having a single config for multiple cds of the same game, which is basically one of the best reason for the m3u format to exist).

A alternative to using m3u filename / using the derived name as config indicator could exist if #6089 is implemented. There the filename is only one of 3 different ways to map a content file to a config from less to more specific. It would be sad if collections lost the ability to get a single config file though (that the m3u loader has).

Anyway, the main risks are in the OP: x68k and files (dumps from dumping groups) which the heuristic can't sort by name. For files which the heuristic can't group but should, there is no actual change (they'd be single), so no 'risk' even if it's disappointing.

x68k has to be handled special already because it has 2 floppy drives so the 'disc control' scheme doesn't work already. Currently i think you can't even change diskettes (there is a way to start with 2 diskettes loaded which serves to start a game, but if you ever need to change, you're screwed).
I don't think it's too much of a problem to blacklist the x68k playlist from this transformation even if the user has it on in the options.

Main problems should be coding the thing. Maybe RA doesn't have access to a regex library and that could sink this idea.

[i30817]

Anyway, @Ferk what do you think of this variation?

[RobLoach]

Could you create a mock up of what you think this would look like? I'm having trouble understanding what you're proposing.

[i30817]

Could you create a mock up of what you think this would look like? I'm having trouble understanding what you're proposing.

Sorry about this delay, but the idea is to make a setting that modifies the behavior of some RA functionality and the game view.

Namely:

1. multidisc games views get turned into a single game view on the games listings by taking advantage of the redump naming convention (like if you could scan m3u). You wouldn't even need a submenu if the rest works correctly.
2. like m3u, you'd be able to make a config for this pseudo file that applies to all cds
3. the cd slots for cd change get filled automatically (but do not change automatically, except...)
4. the tricky part: when loading savestates, remember the cd it was saved on and switch automatically (at least it would be great if this worked) and maybe show all savestates of all discs to be able to go quickly to any part of a multicd game. Especially valuable when autloading on start of game. I don't know if this is possible without rebooting the core on 'switching' from a cd to another through savestate, but even if that is required, it eliminates some UI boredom if it works.

There are many redump games on the ps1 and others that have 4+ cds. I think it would be a good usability gain to have something like this, if m3u scanning as a 'single game' won't ever happen, for the obvious 'filesystem' solution (that tbf, would only be really valuable if 2-4 were the same and those are all runtime features).

If you wanted something visual, I'm not talented at that but it's literally just removing all 'game cd1, game cd2, game cd3...' entries and replace them by 'game'.