[PLAYLISTS] Add an option to use DETECT instead of CRC

[andiandi13]

Many many, many users complains about RetroArch not adding files to their playlists because the CRC doesn't match the original dump.

It can be because of a translation patch, an SRAM patch, and so on...

So I suggest a very simple solution :

Just add an option in playlist menu that could be named 'Ignore CRC while scanning roms' or 'Add unknown roms to playlists', whatever.

The thing is that it'll replace that line

"crc32": XXXXXXXX | crc"

By

DETECT

In order to easily display all roms on playlists, with a label taken from rom filenames, so that roms with a good labels would display thumbnails, and roms with another label (like T-eng 100% etc...) would not display cover (it's not a big deal).

RetroArch playlists are painful to use because of those unique CRC32, and it would be wonderful to simplify it in that way !

For now, I'm creating my Playlists with RetroArch Playlists Manager on Windows, to add my SRAM patched GBA roms, as well as all my non-detected translated roms.

Thanks for reading, I really hope to see that option.

Version

• RetroArch: 1.7.6

Environment information

• OS: Windows, Android, Switch

[ghost]

Why not just use the playlist manager in the desktop menu? It doesn't use the scanner.

Part of the reason why we validate the checksum is because an equal many number of people have bad dumps that won't work and then blame RA or the core instead.

[andiandi13]

@bparker06 What is desktop menu ? Is it the menu that opens with F5 on Windows ?

I mainly manage my Playlists on my Nintendo Switch, I guess there isn't that menu ?

I understand your point, but you can add it as an hidden option that would appear if advanced settings are enable. Advanced settings for advanced users.

Yes there arr bad dumps, but most of players also have goods dumps + patched versions, and it's hard to have those beautiful playlists with all our collection.

[ghost]

One of the problems with this is that there's no way to associate database entries/metadata or thumbnails with playlist entries that don't match checksums, I would consider this a bad UX and defeating the point of working towards a "beautiful playlist".

[andiandi13]

I don't really understood your message, but, why can't RA act like RetroArch Playlists Manager ?

It takes the file name and put it as a displayed title, so that thumbnails can match that name.

e.g. Mega Man Zero (Europe).gba will be Mega Man Zero (Europe), and will read Mega Man Zero (Europe).PNG as a boxart.

Here, it will scan filenames and copy the name as a title in the .lpl file, and refers to file extension and/or current loaded core, to decide which console is concerned and save it as a playlist (e.g. GBA here)

[ghost]

It could, but not everyone's roms are already named properly, and they don't want to rename them. Matching based on checksum fixes that issue.

If the roms aren't named properly then thumbnails won't work. And when you scan without the database, even if the name is correct, it won't know which folder to look in for thumbnails.

[andiandi13]

It could, but not everyone's roms are already named properly, and they don't want to rename them. Matching based on checksum fixes that issue.

That's why I suggested to make it an option and not forcing people to rename their roms. It's just adapting to multiple users.

If the roms aren't named properly then thumbnails won't work.

Yes, but it's a small trade-off for having user friendly playlists IMO...

And when you scan without the database, even if the name is correct, it won't know which folder to look in for thumbnails.

RA can't create a playlist based on the extension ?
For instance well named .sfc files would be added on 'Nintendo - Super Nintendo Entertainment System.lpl' with DETECT lines.

[andres-asm]

Thumbnails shouldn't work like this anyway.
For the packs or on-demand downloads (#8490) forcing filename is fine I guess.

But in the end the playlist format should be able to reference an arbitrary thumbnail filename.
That way on both GUIs we could have an option to link a thumbnail that is not on the thumbnails folder or isn't part of the packs.

The whole al filenames have to match convention was due to the fact that the playlist couldn't grow or store any more information because the format was hardcoded.

The new one has room to grow.

As for the OP.
I figure we could have an option that reads "Add to playlist" then you select a file and it would search the DBs for occurrences with that name, then the user would be able to select the best suited entry.

Of course that's a lot more complex than it sounds

Bottom line, giving infinite hands anything can be done. But that's not the case on this project.

[andiandi13]

As for the OP.
I figure we could have an option that reads "Add to playlist" then you select a file and it would search the DBs for occurrences with that name, then the user would be able to select the best suited entry.

Why not, but it'd be a hundred time longer than manually creating playlists on a PC.

I also thought about adding roms to playlists according to their unique header/informations so that an header can be linked to a name on the database. But I guess it'll require to recreate an entire database.

[i30817]

Here is a intense (but terrible) newflash:

CRCs are already ignored when scanning many many roms types - for serials. Among other things this means that contributing to the libretro-database romhacks CRCs is completely useless¹ and is most of the reason i'm hesitating to continue after contributing quite a lot and creating a tool to auto update my PRs and hacks.

If anything what i want is the exact opposite of this feature: give a option to ignore the serials and go full crc instead of pretending that a serial is a unique id (hilariously, even if you scan something like a snes rom, you get all of its 'versions' and 'dump variants' - and maybe even hacks don't even remember but it may be worse and they simply not be there).

This is hardly unique, since console makers didn't really care often about editions or bugfixes bumping serials, though as i mentioned it gets kafkaesque once hacks come into the picture. I made plenty of suggestions to both make a CRC scanner faster (only do it optionally, only do it for (cd) formats with a inbuilt crc like chd, keep a separate hack database and only check when needed, keep the crcs of cue files for the 'canonical' dump sets etc). The RA project made it clear that the scanner is a afterthought with a rather impressive number of ways it can go wrong and no interest in doing anything to get data the original dump sets didn't provide (filtering mame split sets for only the main 'rom' for instance) so i stopped caring.

¹ in fact i'm about 95% suure that last time i checked, the 'crc' on the playlist files was often bogus now, and a artifact of the old format / code using crcs but the scanner changing to serial as primary key, so you have crcs being 'found' that are completely fictitious.

[inactive123]

Please stop being so negative.

The RA project made it clear that the scanner is a afterthought with a rather impressive number of ways it can go wrong and no interest in doing anything to get data the original dump sets didn't provide

Send us PRs to deal with these issues. The communication couldn't have been clearer. No walls of text are going to fix anything here on its own.

You might be mistaken into thinking we have infinite amounts of contributors, manpower and time to do all this - we don't. We depend on the generosity of coders to help the project forward, and we can't exactly rely on programs like Google Summer of Code either.

[i30817]

I opened the issues for about, oh 1 year? Possibly more. You yourself even mentioned 'uh maybe we shouldn't do this guys' on one of the worst examples and then someone marked the bug as 'minor'.

#7455 (comment)

So no, i'm not going to do your features for you. I just thought it especially precious that people are so misinformed about this that they think CRCs are the 'problem that needs to be solved' when the scanner is a dumpster fire in many other ways because it abandoned unique keys or because there isn't a option to make the scanner stop trying to support everything under the sun and be bad at everything.

I'm not even opposed to fast scan of serials myself, only not when the 'serial' has a obvious other entry in the same database with a different crc. In that case, yes, i'd love the option to 'if it isn't a unique key, fallback to CRC plz'.

[inactive123]

I opened the issues for about, oh 1 year? Possibly more. You yourself even mentioned 'uh maybe we shouldn't do this guys' on one of the worst examples and then someone marked the bug as 'minor'.

#7455 (comment)

So no, i'm not going to do your features for you.

Stop being abrasive and show some respect. Last warning before I get Github moderators involved in here. Any kind of passive aggressive nonsense you can leave at the door from now on, keep it strictly focused on the factual and behave yourself. Nobody has time for this nonsense and this is supposed to be a drama-free developer zone. If you're not a developer, your ability to contribute to this project is limited to say the least, and walls of text don't amount to any code anybody can do anything with.

You say "I'm not going to do your features for you". The way I see it, YOU want us to do all this for you when I have indicated to you already it's not a fact of us wanting to do it or not, it's a matter of not having the necessary manpower for it and resources already being spread thin. Once again, stay polite while you're posting in these threads and don't cause a scene.

Being developers on open source projects is already enough of a burden and thankless job without having to deal with toxic personalities like the one you're portraying right now. Don't make our jobs even harder.

[i30817]

Here's the facts:

RA is using serials, which lead to duplicates. It's also ditching certain emulator specific configurations of roms, lke MAME split sets (which could be supported by just taking the crc of the main game archive and pretend the others are not important). So the project decided to 'canonize' some particular sets, sometimes, and fuzzify most times.

This is still not enough to both fuzzily 'identify' certain games (games are missing), so people are asking for even more fuzzy filenames. I disagree with this because if the 'game is missing' even with serial, there is something else seriously wrong (what? Probably the scanner is dying on a untested hill somewhere on a set. Misidentifying sets of a console for another is my guess, that whole mess of scanner looking at a cd dump 'magic bytes' to find out what type of console it's supposed to be, which is not very resistant to different types of fileformat).

It's also misidentifying game hacks, because naturally they don't bother changing serials.

Finally, RA after getting the first entry of a game list (or the last don't recall) puts its (possibly bogus) CRC on the playlist (not certain, but probably because there are features and code that needs a crc).

Any future feature that depends on crc for uniqueness (say, web data, autoconfigs, auto tweaks, netplay) is built on quicksand.

[inactive123]

I'm not even opposed to fast scan of serials myself, only not when the 'serial' has a obvious other entry in the same database with a different crc. In that case, yes, i'd love the option to 'if it isn't a unique key, fallback to CRC plz'.

My response to this is still going to be the same - make a Proof of Concept PR, if it's good, we can merge it. Our resources are already spread way too thin, and you are seeing armies of developers that are simply not there. This is the realities of an open source project - change happens at home, and if you want to see who is going to be leading that change, look in the mirror for the guy who could likely do it.

[inactive123]

Alternatively, start a bounty and add some money to it so these issues can be incentivized so we can find a developer who would like to take this on.

I wish it were different and I could find a developer overnight who would take this on, but that is not the current-day reality, it was not last year, and it is not the case this year either.

[inactive123]

I'm terribly sorry @andiandi13, but your issue was hijacked by two people in particular who really should have known better and who only seemed to participate in this thread to start drama and air grievances, and apparently caused them to piggyback off each other. To prevent this from spiralling out of control, the post in particular was removed and this thread locked. Please open a new issue and I'll make sure this does not happen again and your issue will be properly heard.

To @grant2258 and @i30817, don't do this again.

[orbea]

Just adding a link to the new issue for reference.

#8624