Integration of ROM Hacks into Online Updater

[SpaceAgeHero]

Dear devs,

I would like to ask for an integration of Rom Hacks (preferably romhacking.net contents) into the online updater of RetroArch.

Here are some additional thoughts:

• maybe somehow like cheats work?
• download complete hack-sets system-wise
• RomHacking.net propably doesn't provide an API so contents must be mirrored / hosted somewhere else?
• ability to enable a specific Rom Hack on demand for soft-patching

Benefit: easier access and use of hacks, like translations, bug fixes, awesome color hacks or other community improvements.

[orbea]

With all due honesty this sounds like a huge maintenance burden and bug farm with very little benefit for most users.

[hizzlekizzle]

I think it's a great idea, but yes, without an API from RHDN, it would be a maintenance burden, for sure.

As for being of little benefit to most users, I think that's what OP is getting at: it would make romhacks more accessible to people who know nothing about them.

[orbea]

As for being of little benefit to most users, I think that's what OP is getting at: it would make romhacks more accessible to people who know nothing about them.

I see what you mean, but do most people even want to use hacks? My impression is that its a niche use case, but maybe I'm mistaken?

I think the biggest problems excluding the lack of an API is making sure the content and patch matches, romhacking is full of patches that only work with very specific roms, some which depend on other hacks or changing the rom from headered and back and forth... (i.e the first patch needs a headered rom and the second patch needs the first patch and a unheadered rom)

I would think this is out of scope, but then I'm not sure we have a clearly defined scope in the first place...

[i30817]

This will probably never happen, but just in case, a aside:

The ips rom hacks for pc-98 and snes that 'require' a header can be made to apply on roms without a header with https://github.com/heuripedes/ipsbehead

bps romhacks of course, never need to care about headers.

btw, this would get significantly more complicated for iso romhacks. Some of them need conversion to iso first and subsequent cue edits, some of them need cdmage image rebuilding etc.

My romhhacks PRs for libretro database list the romhacking.net urls of the patches applied on the romhack entries on the dats in order (multiple patches may be applied) with the idea that all those patches that 'need' a header had ipsbehead applied, so they work with no intro.

I use this project to keep them up to date: https://github.com/i30817/rhdndat

I think the biggest obstacle to this happening is that romhacking.net never bothered to mandate a standard 'container' format. A patch zip might have 2 patch files, ips and bps, 2 ips files, for headered or unheadered, or divide the hacks into multiple options you're 'supposed' to apply at will etc.

[i30817]

I also had the idea of using the urls on the romhack dats on liibretro-database to allow a person to see what 'romhacks' entries are applicable to a original game and opening the urls from retroarch; which is a much lesser version than 'automatizing everything'.

I believe that i gave up on that when i figured out that retroarch s currently not even using the CRC32 of roms to recognize romhacks, so there is really no point to make it easier to apply them when RA won't even recognize them.

[hizzlekizzle]

@i30817 good point on the lack of standards for patch containers. Certainly complicates things.

@orbea I would think the fact that we do CRC scanning would help with the header thing, but it would really require the hacks be pre-sorted to know what goes with what.

[RobLoach]

Softpatching in RetroArch works. Download the patch, and name it the same as your rom file, in the same directory, launch the game. RetroArch will apply the patch in memory.

• Having a UI around ROMHacking.net to download patches would be neat, but as orbea mentioned, it would be a lot of work to maintain.
• i30817 has submitted a few of the most popular hacks to libretro-database/metadat/hacks.
• The Menu UI is difficult to work with, but I could imagine providing users a way to run roms with a selected patch. This would allow users to run a patch on the rom without having to rename it manually.
  

[i30817]

Unfortunately softpatching is not suitable for roms larger than 32mb or so. Retroarch doesn't even support it for the n64 (or NDS... or even genesis plus DX) much less the ps1 and larger.

It's a elegant idea, don't get me wrong (to calculate the crc32 of ips/bps files instead of the final result to match it to a romhack entry). I wouldn't mind creating a new entry that lists the patch crc32 of the patch file for rhdndat output.

However, people use different patch formats than the original for valid reasons all the time (ipsbehead is a example, but not the only one, for example, it's possible to adjust a N64 rom byte order to create a romhack patch that applies to another byte order by applying the patch on the right byte order, switching the byte order and creating a new patch to apply to no-intro for instance).

I also do not keep patch files for cds/roms larger than N64 roms (because they can't softpatch) and instead keep 'revert' patches xdeltas to get back the original after a hardpatch, so i couldn't even calculate those without redownloading all of the cd patches... which i probably won't, for something that is useless to me (and everyone because RA can't softpatch them).

[orbea]

I would think the fact that we do CRC scanning would help with the header thing, but it would really require the hacks be pre-sorted to know what goes with what.

Exactly, also consider that new hacks and translations are added frequently, often not in English. :)

[i30817]

That already occurs. Having a incomplete database is not enough reason to give up having a database, and if people are especially attached to, say, spanish translations, they can PR the database using rhdndat to keep the patches up to date, like I do for many english ones (not all ofc).

I'm really disappointed in RA having the information and not actually using it (or allowing more complete PRs for snes and the nes). Especially when a translation (hardpatched or not) is 'recognized' as the original game because there is no way to turn off the serial scanner for the crc32 scanner.

[orbea]

if people are especially attached to, say, spanish translations, they can PR the database using rhdndat to keep the patches up to date, like I do for many english ones (not all ofc).

Are you volunteering?

[i30817]

No, because i don't know spanish and therefore have no interest in spanish translations (in spite of living right next to spain). I'm just saying that the idea that 'we don't want to recognize translations and hacks because some aren't popular enough to attract a PRs' is bizarre.

[orbea]

You're not understanding my point, the point is that this is massive ongoing project with no willing maintainers.

[i30817]

This - to be clear, just recognizing romhacks, not downloading them - is a optional effort that needs no ongoing maintainers in the main team and there is the possibility to make tools to make it as easy as possible to update them too (my own being a example, though that one needs the roms to calculate the final hashsums). If that's your attitude, you might as well gut everything not from dumper groups from libretro-database.

That there is no way for a user to request to use the CRC32 method again means that those entries that are there, are useless for users.

[ghost]

whats difference of just browsing to romhacking sites like romhacking, download patch and rename as needed? also, how to know if the patch requires a patched rom or not? who is gonna maintain database for this (ugh.. more database files....)

[Lreaper]

It would be really neat if there was a menu option to load content with a specific ips patch, I'm aware softpatching already works if you have the patch in the same folder as the rom file but what if you have more than one patch for one specific game? Having a menu option to load content+patch would allow the user to have several romhacks available for any given game without having to constantly rename/move patches around.

[i30817]

That would require ips/bps softpatching to be externalized from cores, which is something that RA-libretro has steadfastly refused to do for many years.

Though i'm sympathetic about the fact that if they did it would be done badly (by using a tmp file hardpatch) because Windows filesystem / ramdisk apis utterly suck and RA is not willing to make a feature only work on unix or whatever.

About 'more than one patch', i suggest looking into hardlinks in linux, even if they sabotage a trick i'm hoping RA picks up (xattr records of the patched file crc32 for softpatches rom targets).
edit: symbolic links have the same problem... linux should fix this shit...

[Mte90]

I agree that a system to download is not easy but can be a step 2, better focusing on reconizing rom hacks more easily.
Right now softpatching seems cool but what to do in case that the patch are more then one?
Or in the case that the format is not supported by the softpatching?
Also right now renaming the patch doesn't let us to remember what was the patch doing.

An user case like mine:

• Get the rom
• Download different patch
• Try them one by one
• See if it is possible to merge more then one

Right now retroarch use crc that is an issue on detecting some roms so maybe we can find another way to add roms already patched.
Maybe adding in the rom file "Amazing-game.nocrc.zip" and if the filename include nocrc doesn't use the rom scan but add to the playlist of the console (reconizing the file format content).

Another solution can be simplify the adding of patches to https://github.com/libretro/libretro-database/tree/master/dat from the community so the patch are availabl right there so the scanning is more simple.

[i30817]

Right now softpatching seems cool but what to do in case that the patch are more then one?

Modern filesystems can deduplicate roms. When i need this I just create a hardlink, but i bet that some modern filesystems are thinking of doing this automatically. It's more annoying in the case of cd hacks, because those have to be hardpatched and it's in those cases that a hardlink would be great (because huge waste of space). RA doesn't support softpatching after a certain size (enforced by simply not implementing it for the N64 up), so you can't really do anything about it. I simply tend not to have more than 1 patch for cds at a time...

Right now retroarch use crc that is an issue on detecting some roms so maybe we can find another way to add roms already patched.

Merged patches willl probably never be 'automatically' detected. It's simply impossible with any checksum approach. I do put in some merged patches into libretro-database but that's because i merged them manually as good combinations, but even then i don't bother with edgy 'patchers with 10.000 options' that some hackers are doing and just add the 'maximal' option of 'all hacks'. Combinatorial explosions are unfeasible.

Right now retroarch [doesn't] use crc [except on netplay, only that crc is of the game executable so mostly useless for the current database of hacks] that is an issue on detecting some roms so maybe we can find another way to add roms already patched.

FTFY. This regression was added because people were complaining that their self-dumped games in random formats were not being recognized by the scanner and were slow to scan so now your 'reward' is that all game hacks are scanning as the original game because the main scanning method was replaced by something not using complete ROM checksums and thus imprecise. To solve this All Roads Lead to customization of the scanner and amortization of the checksums.

From the lack of even response, a issue that all RA contributors are not interested in btw.

Another solution can be simplify the adding of patches to https://github.com/libretro/libretro-database/tree/master/dat from the community so the patch are availabl right there so the scanning is more simple.

'Just' pressure romhacking.net to release a good dat or do what i do and use a tool to keep your local patches up to date and submit the updates to libretro-database (even if the database is currently not used by RA, the tool is still useful to detect when i need updates). Though be prepared to get your commits rejected because fears that 'too large' a patch collection will slow the scanner (can't commit the checksums i have for NES or SNES here). Though it's arguably irrelevant anyway, since they're not being used.

And you probably are having a 'XY problem' moment here anyway. It's not the lack of libretro-database data that is sabotaging the recognition of hacks in RA (though ofc there are many patches/games/systems i'm not interested in so never contribute). It's the scan method (imprecise console specific serial vs precise checksum).

[Imrhien]

All these ideas are cool but definitely hard to implement.

Short-term - why not implement a way that the RA romscanner can still add unrecognised games to the playlist?

I'd be happy with RA just parsing in the filename of the unrecognised ROMs and adding them to the playlist. Perhaps all the unrecognised titles go into a separate playlist (as RA would have no way to know which system to associate them with).

[i30817]

It has that - today anyway, jdgleaver introduced it after 2019 - it's called 'the manual scanner'. It's not as 'fire and forget' as the usual scanner but it's infinitely more usable if you're not simply using no-intro and nothing else (including hacks) and can tolerate to scan per platform once (you can rescan with the same settings from the playlist menu).

If you want images with your games, might i recommend my own tool that uses fuzzy matching (of the game name in playlists) to attempt to match it to the libretro database filenames? Obviously, this won't work for hacks often unless they're actually translations and you don't translate the original name - or if the libretro thumbnail server simply doesn't have the thumbnails for that game, which i was surprised to see was a common issue (worse, if it doesn't have that, it's likely to get a game confused by a sequel or prequel, ie: dead or alive 2 getting the dead or alive thumbnails etc).

https://github.com/i30817/libretrofuzz

if you're trying to make the scummvm core work, a unfortunately terrible usability task, i also did this: https://github.com/i30817/libretro-mkscumm

It's still not easy, but at least it gets you the names scummvm actually has the names as, and you get at least some images (i've been adding the games i have, which are not complete by all means - not enough other languages versions, no patience to create symlinks to PR them all) but it's kind of in ok condition with that other tool for the images for me at least.

Also there is a issue of correctness in actually following the description. Let me explain, the RA readme of the scummvm core mentions that you 'should add your games inside the core before using the scummvm.ini file [ids] to identify them in playlists'. What they don't mention is that creating this scummvm.ini file is actually very very necessary sometimes because the [ids] often don't identify everything.
Sure there is a root engine 'id' but starting a game with the 'root engine' is often a very different result than starting them with the parameters that the scummvm scanner detected for the specific version of the game. Moreover that scanner sometimes adds enhanced versions of game entries in some engines (nuvie and bladerunner in my experience).

So the problem is that 'lazy' people 'just' go to the scummvm site, look up the game they think they have and add that 'master engine id' into the .scummvm file, completely ignoring any flags that particular engines might add for some particular version or language detection/choice that the scummvm scanner would present. It was my aggravation with doing this hundreds of times - from a large, but not comprehensive scummvm game collection, even if most engines are not that polished - that made this tool a necessity for me - and i was still doing it wrong, so you can probably see how this is total nonsense to expect most people to go through.

[i30817]

Btw the situation is slightly better today because a outside contributor coded the softpatch part of RA to accept multiple patches at once, so now if you want that, you don't need to create a merged patch (tiresome), just add '.ips1, .ips2' after the main one.

Well, except in the unfortunately common cases where the patch has some special needs for the source rom, like a header or a iso instead of the standard redump cue dump etc. That last wouldn't work anyway, something which has not changed is that the softpatch engine in retroarch still doesn't accept xdelta and doesn't work with large isos. Or any isos. Or any format that is larger than some number of megabytes i don't remember.

btw for snes and pce headers, this utility is useful https://github.com/heuripedes/ipsbehead

And works almost all the time (if it doesn't work it's because the original ips was doing something wrong and fucking with the header somehow, which only happened once in several dozen patches - i didn't test it when i didn't actually need it because i'm no that foolish, so maybe that breaks something, although from the readme description it seems to me it could be avoided by simple auto-detection ).

[mdsmendes94]

That would be an amazing feature to have for sure, RetroArch integration with RHDN or MetroidConstruction or any other host that provides romhacks

Neither RHDN or MetroidConstruction seems to have an api, although SMWCentral seems to have one ready

So if I understanding this correctly something like what the OP is asking can be done, although for now only for Super Mario World for the SNES?