Is Ludo replacing the retroarch scanner too? #203
Comments
|
The scanner of Ludo is even more rudimentary than the one of RetroArch, as one of the design goal is to keep the codebase small. Discussing these two proposals at the same time could become chaotic, so let's focus on the 1st one here, and we can then discuss the second proposal on a separate thread. I have two questions about the extended attributes:
|
there is a equivalent on NTFS but i don't know anything about it, since i haven't touched windows in 2 decades. They are, of course, not compatible if you move the filesystem between OSes. Extended attributes are compatible with macosx, freebsd etc though, though again, i don't know how much of that is good publicity, since i don't use them.
Not necessarily, i just find it attractive to me because it allow me to tie in the idea of attributes for checksums to softpatching - ie: in my tool, the 'user.rom.crc32' refers to the final patched file, even if it's a softpatch, which is something RA has problems with and which requires more byzantine code to handle (when a patch updates etc). You could easily write the scanner results to the files extended attributes for 'more of the same, but with a persistent cache now', it's just a waste imo. |
|
It's an interesting way of tackling the translation patch problem yes. I had another plan for that: building and hosting a patch store. (I think it is legal to host patches) However, Ludo doesn't even have the soft patching logic implemented. I've looked into it quickly, it's a medium amount of work. So the first step would be to support soft patching. I wasn't aware of the problem of ROMs being modified by the emulator. In what cases does this happen? |
|
All the computer floppy/hd cores, except sometimes they already adapted a adhoc 'copy on write' strategy to prevent this. Recently dosbox got a VFS of some kind because steam wants their game dirs to be read only now, though of course RA doesn't know about it. X68k still modifies its hd checksums as soon as you boot (DOS tmp files), the Amiga, etc. I think the famicon console and some older ones had a dedicated batteries for writable parts of cartridges that are either not necessary on emulators or since been using transparent copy on write from the emulator side that no one remembers they're not actually 'ROMs'. Until the main emulators either adopt a strategy like the copy-on-write or 'record it somewhere that moves with the file' like i want here, creating a X68k .dat is somewhat of a academic question. I 'just' created a script (see the dosboxmount-src dir there) to mount my actual game drive on a COW filesystem when i want to play DOS or x68k. It comes in handy because my actual game drive is read only too (compressed). Then i 'only' have to remember that a RA playlist for dos (for instance) needs a 'launcher app' before starting... I'm not enthused about this solution because it's very easy to get out of sync and hard to resist the temptation to just mount another dir (i probably should have coded the script to be more flexible about mountpoints dammit). I also strongly dislike that the DOS dumps require the dosbox.conf to be in the same dir as the main executable to be 'portable' (from my understanding, this is basically a dosbox hack to not have to have some kind of mapping). TBF this is probably because of another hack, namely DOSBOX dir separators inside conf files are not portable if you're trying to mount a native path. You can imagine the mess when a game want to not be the root of the drive or something - mount the game.exe based on the dat, run the dobox.conf in the same dir automatically, try to mount the native path with '..' because a '\' is too much trouble to automatically translate, mount the cds inside the game dir, etc. I know of one DOS game that is even worse than normal and keeps saves and config options on the executable file itself with selfmodification... (it's the adventure game Hook btw). |
Another thing, this is not always true (being 'legal' that is), and besides that hacks are getting larger and larger and sometimes more complex to apply (thank compression and cryptography for that). No emulation project - rightfully - wants to download 1 gb zips that when you open have 'first use $XCONSOLE_TOOL to sign the data with your private key'. But more power to you if you manage something respectable with this. You'd have to mandate some pretty strict standards imo. For one all game patches for the snes should be without headers (easy enough with ipsbehead). Then '1 patch per archive' and 'standard named patches / readme's / formats' rule (much harder to herd cats), other problems i can't foresee. This is basically the same problem as distribution of game mods.... even standardized stuff like WEIDU (for infinity engine games) and ESPs (for Elder scrollls) gets sabotaged by the moddders insisting on $WEIRD_INSTALLER_FORMAT with essentially aleatory directory structure or neglecting to invest into tools to handle dependency order. Multiplatform portal for something as diverse as romhacking formats would be much more chaotic unless you were extremely strict, to the point 'doesn't pass my (on submission form to make them care) validation tool, your patch doesn't get in' would probably be the only way - and that would be a shadow of the content in romhacking.net, that already doesn't have everything. |
|
Uh sorry about the rants/derails when you said it would be too chaotic to discuss both ideas at the start. I can delete them if you want, the negativity isn't helping much there. I'm sometimes not the best at self-awareness and get too excited about 'why this is bad' instead of 'how to make it good' |
|
No, keep them, it's valuable information for future developments |
|
I close this for now because it is too advanced for a frontend like Ludo |
If so i'm interested in proposing two 'extensions' to the scanner concept in retroarch that i find it unlikely xmb is going to support because they'd require complex dependencies i doubt will work on console ports and the first won't work on windows.
libretro/RetroArch#8873 <-take advantage of extended attributes to get a 'stand in' for retrieving the crc32 instead of calculating it (like retroarch already does for zip files). As explained in the issue this also solves the case of softpatches crc not matching the final crc32 the core sees (if you use my tool to create the xattributes) and 'solve' the case the roms that the emulator writes to in normal operation and therefore change its checksum much later when you try to reconstruct the playlist (though i prefer to use system copy-on-write to keep the original rom, it's also helpful there, because you can still rescan the 'altered' copy on the copy on write). In fact extended attributes are great because they move with the file.
libretro/RetroArch#8672 <- give the user complete control over the scanner behavior with files on the rom dirs filesystem. Helpful to pass launcher files to cores that need it, disassociate those launcher files from the roms that have the actual crc but still use it, being faster by whitelisting the types the scanner scans per directory subtree, allow to use multiple types of scan if you implement them, etc.
It's a complex idea, so it needs to be read (both posts), but the tl;dr is : users that don't care can use the dumb scan and get false positives (because RA is scanning file cues when it should be track 1's because it needs cues to get a complete index file, or is using serials that misidentify hacks as the original game etc) or waste a lot of cpu power crc32'ing things they don't need. Users that care organize their games per console / required launcher file, and drop scanner control files into the file hierarchy.
The text was updated successfully, but these errors were encountered: