RFE: new optional field(s) for per-game configurations

[i30817]

First and foremost consider this:

• Only RetroArch bugs should be filed here. Not core bugs or game bugs
• This is not a forum or a help section, this is strictly developer oriented

Description

Idea here is to make game configs less hostile to distribution. If these fields are filled, retroarch would replace looking for a config of the same exact name as the game entry point to the core (normally .cue, iso, dirs, m3u etc, what goes on the playlist), with something that the scanner already collects and then match the file it found that id to the game or software.

For instance there could be:

id-md5 = 9cf2f56ea9c4824fdd3b6b4053bc02be
id-serial = SLPS-Whatever

Any one might exist or none (some platforms don't have serials and would lose the important 'fuzziness' property until some replacement comes along). Could, maybe should, be added on creation of the config file (if possible).

To make multiple files could to point to the same game let one 'win', RA would chose the config with the most specific id.

In effect, this works by taking a set of files, iterating over them and choosing the first id-md5 that matches the game data. If match return. If none, iterating over all id-serial and choosing the first that matches. If match return. If not, iterating over config filenames and choosing the first that matches (current behavior).

This is done to allow a more specific md5 to always override the more fuzzy serial or filename. But never vice-versa. It would prevent distributors of configs from having to care about the names of their configs matching with names of their dumps (indeed, they're often wrong) and by deleting the id-md5 broaden the usability to any dump that the serial can be parsed from.

Expected behavior

That there would be some effort to make it easy to distribute configs currently without user effort. The flexibility of several levels of match from 'dump specific' to 'works on hacks' is also a important property. Filename match is not enough since it requires user effort and stymies effort of creating large collections of 'ideal' configs (to remove blackspace without stretch is the usecase that already exists). Naturally there is a almost infinite number of possible ids for specific systems (see the sega header md5) but at least the 'serial' should be good for RA because it already had a fast serial scanner that worked on multiple kind of dumps... supposedly, i didn't read the code.

Coincidently or not, i just finished a tool to extract serials (ps1, ps2 and dreamcast) and sega header md5 sum from any dump by analyzing the track 1 in libmirage.
https://gist.github.com/i30817/f58c7f904fe44c4599a6c9d48a60bce0

Possible problems i can see:

1. Parsing serials from arbitrary dumps is difficult to do portably. Indeed, the script i posted above is not portable, because it's using libmirage (part of the linux cdemu cd emulator) to parse out 'track 1' off any cd dump type that cdemu supports. Even if you have a good library like libmirage, it's unlikely that some of the more NIH copy protection made formats get supported by it. Gamecube/wii discs for example for the dolphin core. However, libmirage is 'supposedly' extensible with arbitrary text parsers so it's not impossible (i even managed the data i wanted on dreamcast discs - granted it was at sector 0 - in dreamcast track 1 is not complete and needs to be joined with the last for LBA access or some crap like that, so i didn't actually manage to chuck track1 to a iso parser in that case). RA supposedly already could parse serials though, for the old style scanner.

2. False positives: imagine a hack that changes a game resolution, suddenly the 'redistributed file' that is matching for serial, is ruining the image not fixing it (or any setting that the hack doesn't like).
   Very low risk, since these kinds of hacks are very few. Can be fixed by the local user deleting the config (which might be difficult with random names), the user creating a new config (new id-md5 would override) if that's possible, or even better the distributor creating a new file for the hack with the md5sum of it and keeping the 'normal' config. This is a very deliberate decision if the config creation saves both ids, because to distribute configs without id-md5, the distributor must edit the file and probably understand that he's broadening the number of dumps and hacks it applies to.

3. If a 'lazy loading' usage of hashcode field of the metadata is adopted like i'd prefer - ie: asking the user wait a bit to proceed when it's used -, creating those fields automatically on 'saving per-game' configs, will be annoying since it's not a ideal time ask the user if 'do you really want to save your configs, this will take a long while?' Instead, maybe each 'step' on the match iteration should be disabled if the data is simply missing and was never filled (ie: manually created playlists etc).

4. some platforms might not be easy to parse serials for, or there might be some identifier that could work for the same purpose but it's not exactly a 'serial' (sega headers come to mind). Since the risk is that platform specific game id not existing, this is not a problem except in as it makes the feature null and void. But i suggest possibly replacing 'serial' by something else that identifies any game (if not 'uniquely' any dump) to keep the feature working. It's possible some platforms would put in something different on the 'id-serial' or the names could be more neutral. 'id-dump' and 'id-fuzzy' maybe?

5. I don't expect a 'huge' speed even if the scheme does require all the per-core configs (or maybe just all configs) loaded, and the game metadata loaded, in order to run the match algorithm.

6. Maybe instead of matching and returning on the first, it iterate on all levels to warn against duplicate files (ie: two or more of same id-md5, two or more of same id-serial, (on windows) two or more of the same name (but different case). Dunno if it's worth it.

uh, dunno anymore problems. I guess RA might not want to scan serials or the header if you went to the trouble of replacing those kinds of imprecise scans by md5sum of whole files? The fuzziness is sort of the point here though.

Actual behavior

No effort towards this

Steps to reproduce the bug

Not a bug

Bisect Results

[Try to bisect and tell us when this started happening]
Forever

Version/Commit

Not applicable

Environment information

Not applicable

edit: edited dumb algorithmic confusion on my part.

[andres-asm]

You can't leverage playlist info for this, playlists are volatile. Also you can rely on a the scanner, that would mean scan the file on every load.

…

On Fri, Jan 12, 2018, 6:46 AM i30817 ***@***.***> wrote: First and foremost consider this: - Only RetroArch bugs should be filed here. Not core bugs or game bugs - This is not a forum or a help section, this is strictly developer oriented Description Idea here is to make game configs less hostile to distribution. If these fields are filled, retroarch would replace looking for a config of the same exact name as the game entry point to the core (normally .cue, iso, dirs, m3u etc, what goes on the playlist), with something that the scanner already collects and then match the file it found that id to the game or software. For instance there could be: id-md5 = 9cf2f56ea9c4824fdd3b6b4053bc02be id-serial = SLPS-Whatever id-sega-header-md5 = 47235d80682efdd72b098083f76ea7dc All of these would be permissible, but none is obligatory. Could, maybe should, be added on creation of the file (if possible). One wins the 'match', probably best from most specific to least so: id-md5 wins over id-serial (this one is interesting for fuzziness because it would work for most dumps or hacks), wins over sega header (same as serial but less recognizable) wins over filename matching RA currently does. Expected behavior That there would be some effort to make it easy to distribute configs currently without user effort. The flexibility of several levels of match from 'dump specific' to 'works on hacks' is also a important property. Filename match is not enough since it requires user effort and stymies effort of creating large collections of 'ideal' configs (to remove blackspace without stretch is the usecase that already exists). Coincidently or not, i just finished a tool to extract serials and sega header md5 sum from any dump by analyzing the track 1 in libmirage. https://gist.github.com/i30817/f58c7f904fe44c4599a6c9d48a60bce0 Actual behavior No effort towards this Steps to reproduce the bug Not a bug Bisect Results [Try to bisect and tell us when this started happening] Forever Version/Commit Not applicable Environment information Not applicable — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6089>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABpC0IfZ4NxO-IzdoimsyUG1ehvFlIN1ks5tJ0X8gaJpZM4RcLt9> .

[i30817]

Why shouldn't you rely on playlists to store this game information? I presume this already happens somehow since the game gets scanned once and only once.

So i don't see the problem with the playlist being 'volatile', if it doesn't matter for other uses? Like netplay I imagine. There is still the fallback to filenames that would still work for user created files i imagine, even without the playlists and without re-scanning.

On another matter, If there is a disadvantage in decoupling the names of the configs from the game it is associated to it's that it makes it unintuitive to find, or maybe very hard indeed (if the gui doesn't name it). This might cause a situation where there are several versions of configs for the same game on the configs dir because the user just merged several 'compilations'.

Which is suboptimal, i'll admit. Maybe the 'first chosen' shouldn't be 'first' but the first of a sort of config file creation date. And maybe have a option to delete all other duplicates (of the same 'level of id')

[i30817]

BTW, two issues i opened on cdemu sourceforge page says they fixed gc and wii isos and are 'maybe' on sega gd-rom (sega gd-roms have a segmented track that needs to be 'cheated' to be read as a normal dump - i think that's the issue, didn't pay too much attention to the details).

Not that this is exactly necessary on this case for the dreamcast because libmirage reads track1 sector 0 of the gdi files fine fine (if libmirage happened to support gdi, i tested it out with toc+bin from a extracted .chd - iterating over all sectors went out of range though as expected), and that sector contains the game id/sega header. But complete gdi support would at least allow gdi files to be recognized... though i don't know if that would be useful for emulator cd mounting, irrelevant for this purpose anyway.

So if you want to use libmirage (if you don't already lift serials from all kinds of dumps), making it portable is a approach. Dunno if it's actually possible, but i think so, because the part of cdemu that is a kernel driver is separate from libmirage according to their UML on the site. It 'unfortunately'* uses GLib and GObject, so i doubt it can be made portable to other OSes without forking. But i really don't know.

*it's actually fortunate for me, because it makes using it on python easy.