[BREAKING] A Smart Scraping system to allow for standardized scraping metadata and inferring information from only the filename and file data. (Fixes #167)

[chyyran]

This pull request breaks the IScraper API and removes the IIdentifier API

This pull request adds 3 new interfaces and changes the Snowflake.Scraper.IScraper API

IStructuredFilename
Implemented as Snowflake.Romfile.StructuredFilename

Represents a structured ROM filename using either the TOSEC, GoodTools or NoIntro naming convention. This provides scrapers a standard way of getting a game's title and related metadata such as region code and year.

IScrapableInfo
Implemented as Snowflake.Romfile.ScrapableInfo

This data is passed to the scraper plugin and contains information such as a queryable title and game ID if present.

IFileSignature
Implemented per platform in Snowflake.Romfile.FileSignatures
This interface provides a contract by which certain platforms are able to infer and determine a game ID and platform.

IScrapeEngine
Implemented as Snowflake.Service.ScrapeEngine
This interface provides a service is able to create an IScrapableInfo given the ROM filename by using FileScrapers to determine the relevant platform and by extracting any information from the filename itself. It is also able to calculate the best result given a set of scrapers and automatically use the most accurate one.

[chyyran]

Waiting on #171 as we don't want to change ICoreService every time we add a new service.

[chyyran]

Merged master into this

[chyyran]

This PR should also encompass the deprecation of PluginPreferencesDatabase, or atleast using it to determine scraper order. Instead use a graceful fallback system, if one scraper fails use the next most accurate as determined by a accuracy_weight value int in the plugin.json file. The scraper engine should also be able to determine the likeliness of the scraped value. Although quick scraping is desirable, accuracy takes preference over speed.

[chyyran]

FIleSignatures left to write

• NINTENDO_GB/NINTENDO_GBC
• NINTENDO_GBA
• NINTENDO_SNES
• NINTENDO_WII look for 0x5D1C9EA3 at 0x018
• NINTENDO_GCN look for 0xC2339F3D at 0x01C
• SEGA_GEN
• SEGA_CD
• SEGA_32X
• SEGA_GG

We can probably also identify .NES files by the iNES header.

Any other types we will have a method that matches file extensions to rom type for best-guess

[chyyran]

This isn't going to build because I'm working with a FileSignatureTester project that I'm using to test the filesignature plugins locally, which I'm not going to commit, so until I've written all the FileSignatures and cleaned it up it won't build.

[chyyran]

I don't usually rebase commits before a merge, but since this is shaping up to be a huge PR, I'm putting down a reminder to squash similar commits here.

[chyyran]

The N64 FIleSignature comparator uses @bryanperris's code Soft64.IO.ByteSwapStream, and is licensed under GPL thusly. Perhaps adding a special thanks section to NOTICE would be in order due to use of third party code in these FileSignatures (SFOSharp as well)

[chyyran]

• NINTENDO_SNES
• NINTENDO_NES (iNES) at beginning of file.
• SEGA_GEN
• SEGA_CD
• SEGA_32X
• SEGA_GG

[chyyran]

LGTM after some unit tests and integration testing. The javascript bindings will also need updating as well as scrapers.

[chyyran]

I'm porting the existing scraper plugins to the new API and writing some tests. To be honest this PR has been in the queue for too long so I'm hoping I can merge by next week once I've confirmed everything's working.

[chyyran]

We're in business 👍

[chyyran]

Just have to write tests and this will be merged

[chyyran]

I've looked over it and since most of this stuff deals with plugins it will be difficult to properly testing it unit-wise. I'll merge and write tests for IStructuredFilename later.