[Feature Request] Hybrid Scan (database with manual fallback)

[hizzlekizzle]

Description

Currently, you can do a standard database scan or you can do a manual scan that bypasses the databases. If you have games that don't match the databases, you have to do a full manual scan because there's no way to easily discern which games were identified successfully against the databases and which were skipped (i.e., to come back through and manually scan them).

Ideally, when a file does not match against the databases, it would fall back to the manual scan method for that one file and then return to the database scanning for subsequent files, so everything makes it into a playlist and files that match the databases gain the benefits for a proper scan.

Expected behavior

Files are matched when possible and simply added by name if not.

Actual behavior

Either they get scanned and missed (database scan) or they get added without the chance of a real match (manual scan)

Steps to reproduce the bug

1. Take a directory full of files, some that match our databases and some that don't
2. scan 'em

Bisect Results

always

Version/Commit

You can find this information under Information/System Information

• RetroArch: 1.8.9

Environment information

• OS: linux
• Compiler: gcc v9

@jdgleaver : I don't expect you to work on this, but do you know of anything that makes this unfavorable and/or impossible to implement?

[QuaiGoner]

Really needed feature

[jdgleaver]

@hizzlekizzle This can never work in the way you intend, for one simple reason: if a file doesn't match the database, then it is impossible to know the name of the playlist to which it should be added! So the current database scanner simply cannot handle non-matching files.

What we can do, however, is add a 'search database' option to the manual scanner. So if the user provides a system name corresponding to a database, then it will check for a match as each file is scanned and grab the info if it can, or just fallback to the current behaviour if not. Once this feature is in place, I would be inclined to depreciate the existing database scanner, and promote the manual scan as the 'default' option.

I can certainly put this on my TODO list. I won't give an ETA, though, as I'd need to have a proper look at the database code before attempting to start any sort of implementation...

[bslenul]

Maybe it could be implemented in the directory scanner like this:

• You go to Import Content > Scan Directory as usual, but instead of just selecting the folder, you now have 2 options:
  • "Use Specific Database File" with 2 choices: "OFF" (default) or pick an existing database file in the list.
  • "Content Directory": where you select the folder and start scanning.
• If you leave "Use Specific Database File" OFF, it will act as it currently does: scan and check db files, ignore unknown entries.
• If a db file is selected however: scanner will search ONLY in that db file and scanned files will go to that playlist ONLY, unknown entries will be added to that same playlist using their filename, just like the manual scanner.

So we'd have the directory scanner for a "quick and simple" hybrid scan if a db file is selected, and the manual scanner for a much more precise scan.

Mockups to get a better idea:

[RobLoach]

This PR is related, as it improves the serial scanning code: #10291

[hizzlekizzle]

@jdgleaver Ah, yes, that's a very good point about not knowing the playlist. It occurred to me after I made the ticket and I went over a few possible solutions in my mind, but none of them were good lol

I think your proposal makes for a good, intuitive workflow, making the most widely desirable behavior the default and then whittling that down if required on an individual basis.

I agree with deprecating the old process, too, as I can't really imagine any situation where it's better (though I'm sure someone will tell us noisily about it after it's deprecated...).

I appreciate you wanting to take it on. No hurry, of course, as I know your plate is quite full already.