(Database) Serial scanning for Gamecube/MegaCD/SegaCD/Saturn/PSX/PSP/Dreamcast/Wii

[pkos]

These updates improve support for "Directory Scan". While previously the scan was made against raw disc serials, this code converts and parses those serials into redump.org curated serials.

Features

• Logs and playlists now include these Redump serials
• Multi-disc support
• CRC checks are still enabled and disabling them may increase performance
• Bad discs are scrubbed and logged

Related Issues

• @RobLoach has updated the libretro-databases to include serials (3/17/20).
  Add serials libretro-database#1026
• @RobLoach has updated some string parsing to redump codes. This added support for Playstation - Portable as confirmed no changes to the existing psp detect serial function was needed. Clean the serial RobLoach/libretro-dats#37
• @RobLoach has updated muti-disc appendices (3/28/20). This update added support for multi-disc scanning. Append the disc number to serials libretro-database#1032
• Fixes (database) add support for another Korea PSP code #10282
• Database-related issues...
  • Index the Redump Serials... For all disc-media, including Dreamcast libretro-database#990
  • Redump Sega - Mega-CD - Sega CD missing serials libretro-database#839
  • Missing PSP Games in Database libretro-database#813
  • Saturn, SegaCD and Dreamcast using CRC instead of Serial libretro-database#613
  • PCE CD, Sega/Mega-CD and Saturn Games will not scanned correctly libretro-database#423

Currently, scanner will not detect .gcz, .cso files, but it will crc them.
#7428

This PR code replaces changes already merged here:
Reworked magic numbers.
#6544
#6555
#5239
Serials are in playlists.
#6548

Related Pull Requests

The following libchdr issue will crash the scanner.
#10368
Issue resolved (3/31/20)
#10376

Reviewers

@realnc
@celerizer
@Alcaro
@panasonic
@phcoder

They have added support in #programming, fixing bugs and providing C code.

[pkos]

Originally here, but rebuilt on RA 1.9.0.

#10291

[inactive123]

Cool work! I will ask @jdgleaver when he has more time (probably after the build infrastructure proceedings) to look into this. Rest assured, we have a high degree of interest in this and will get back to you!

[RobLoach]

Just tested and it scanned the games I tested on Dreamcast, GAmeCube, Sega CD, PlayStation, PSP, Saturn and Wii.

[pkos]

There is a bug preventing the insertion of the serial into a playlist. Working on that bug.

[pkos]

Tests

Sega - Dreamcast
1270/1356
Sega - Mega-CD - Sega CD
420/533
Sega - Saturn
2195/2326
Sony - PlayStation Portable
496 - inconclusive stalled on crc32 / 2150
Sony - PlayStation
9974/10422

This is actually about 95%.

[pkos]

There is a bug preventing the insertion of the serial into a playlist. Working on that bug.

Fixed in latest push. Playlists now contain proper crc32 or serials.

[lgtm-com]

This pull request introduces 1 alert when merging b735658 into 94beed5 - view on LGTM.com

new alerts:

• 1 for Empty branch of conditional

[kivutar]

This PR looks very good. We should merge this.
Are the travis errors related to the changes? Some of them look unrelated, they happen in graphics parts of RA.

[pkos]

Looks like the conflicts are resolved.

[pkos]

Moved string functions to libretro-common

[pkos]

Moved string functions to libretro-common

[lgtm-com]

This pull request introduces 15 alerts when merging 5485a64 into 9e84c5c - view on LGTM.com

new alerts:

• 13 for Lossy pointer cast
• 1 for Empty branch of conditional
• 1 for Implicit function declaration

[pkos]

This PR is ready to merge.

[Sanaki]

This has to do with scanning, not playing. I have no knowledge of openemu at all, so I'm not sure how it could tie in. I don't know if it's even libretro-based.

[pkos]

The last commits resolved the LGTM alerts/overflow. With 0 alerts/errors it should now be ready.

[RobLoach]

This is looking really good! Will do more testing on it this weekend.

[eku]

Any progress on this PR is welcome?

[ofry]

@pkos Please fix @RobLoach reviewed style nits.

[ofry]

@pkos Now there's merge conflict too.

[pkos]

Ok, due to automated workflow actions that were recently enabled, the crowdin.yaml file was corrupted in my PR. To fix this I reverted several pushes and force pushed the original crowdin.yaml file from RA master. The PR has been compiled, tested and is bug fixed and in working order with all the latest change requests.

[pkos]

Finishing up now

[RobLoach]

Did another test of this and it's looking really solid. Picked up files that I had from GC, Wii, Dreamcast, Sega CD, Saturn, PSP, PS1, and PSP(PSN). A couple code suggestions, addressed in pkos#1 .

[pkos]

Whitespace may exist, however in this diff no whitespace changes are marked. We've removed and replaced them at this point.

[pkos]

"star wars"

Amstrad - CPC | 7/7 | dsk
Arcade (MAME 2000) | 1/20 | rom
Atari - 2600 | 5/5 | a26
Atari - 5200 | 2/2 | a52
Atari - ST | 0/6 | st
Atari 8bit | 0/2 | bin
Coleco - ColecoVision | 1/1 | col
Commodore - 64 | 1/7 | tap
Commodore - Amiga | 7/7 | ipf
Mattel - Intellivision | 1/1 | int
MS-DOS | 4/9 | exe/com/bat
Nintendo - Game Boy | 6/6 | gb
Nintendo - Game Boy Advance | 16/16 | gba
Nintendo - Game Boy Color | 4/4 | gbc
Nintendo - GameCube | 0/38 | gcz
Nintendo - Nintendo 3DS | 0/13 | 3ds
Nintendo - Nintendo 64 | 16/16 | z64
Nintendo - Nintendo DS | 26/27 | nds
Nintendo - Nintendo Entertainment System | 6/9 | nes
Nintendo - Super Nintendo Entertainment System | 19/19 | sfc
Nintendo - Wii | 0/20 | gcz
Panasonic - 3DO | 2/2 | chd
Sega - 32X | 21/21 | X32
Sega - Dreamcast | 3/4 | cue/bin
Sega - Game Gear | 3/3 | gg
Sega - Master System - Mark III | 1/1 | sms
Sega - Mega Drive - Genesis | 1/1 | md
Sega - Mega-CD - Sega CD | 8/8 | chd
Sharp - X68000 | 7/7 | dim
Sinclair - ZX Spectrum | 7/7 | tap
Sony - PlayStation | 4/44 | chd
Sony - PlayStation 2 | 17/88 | cue/bin
Sony - PlayStation Portable | 0/19 | cso

[pkos]

"disney"

Arcade (MAME 2000) | 0/4 | rom
Atari - 2600 | 1/1 | a26
Commodore - Amiga | 7/7 | ipf
MS-DOS | 21/21 | exe/com/bat
NEC - PC Engine - TurboGrafx 16 | 2/2 | pce
Nintendo - Game Boy | 43/43 | gb
Nintendo - Game Boy Advance | 95/93 | gba
Nintendo - Game Boy Color | 38/38 | gbc
Nintendo - GameCube | 70/71 | gcz
Nintendo - Nintendo 3DS | 0/52 | 3ds
Nintendo - Nintendo 64 | 25/25 | z64
Nintendo - Nintendo DS | 136/140 | nds
Nintendo - Nintendo Entertainment System | 24/42 | nes
Nintendo - Super Nintendo Entertainment System | 73/73 | sfc
Sega - Dreamcast | 13/16 | cue/bin
Sega - Game Gear | 46/46 | gg
Sega - Master System - Mark III | 12/12 | sms
Sega - Mega Drive - Genesis | 56/56 | md
Sega - Mega-CD - Sega CD | 3/4 | chd
Sega - PICO | 50/50 | md
Sony - PlayStation | 50/227 | chd
Sony - PlayStation Portable | 0/46 | cso

[pkos]

added prefix to PSX logic

|| (string_is_equal_fast(raw_game_id, "SCES_", 5))

test "disney"

before
Sony - PlayStation | 50/227 | chd

after
Sony - PlayStation | 150/227 | chd

[pkos]

"marvel"

Amstrad - CPC | 4/12 | dsk
Arcade (MAME 2000) | 54/123 | rom
Atari - 2600 | 1/1 | a26
Commodore - 64 | 0/14 | nib
Commodore - Amiga | 2/2 | ipf
MS-DOS | 18/14 | exe/com/bat
Nintendo - Game Boy | 7/7 | gb
Nintendo - Game Boy Advance | 42/42 | gba
Nintendo - Game Boy Color | 11/11 | gbc
Nintendo - GameCube | 0/52 | gcz
Nintendo - Nintendo 3DS | 0/29 | 3ds
Nintendo - Nintendo 64 | 1/1 | z64
Nintendo - Nintendo DS | 75/76 | nds
Nintendo - Nintendo Entertainment System | 6/8 | nes
Nintendo - Super Nintendo Entertainment System | 31/31 | sfc
Sega - 32X | 1/1 | X32
Sega - Dreamcast | 8/8 | cue/bin
Sega - Game Gear | 19/19 | gg
Sega - Master System - Mark III | 4/4 | sms
Sega - Mega Drive - Genesis | 37/37 | md
Sega - Mega-CD - Sega CD | 3/3 | chd
Sega - Saturn | 8/18 | chd
Sony - PlayStation | 44/62 | chd
Sony - PlayStation 2 | 8/108 | iso
Sony - PlayStation Portable | 0/35 | cso

[pkos]

added conditional to fix raw "-5" and "50" suffix for Saturn:

     if (strcmp(check_suffix_5, "-5") == 0 || strcmp(check_suffix_50, "50") == 0)
     {
        strncpy(rgame_id, &raw_game_id[2], length - 4);
        rgame_id[length - 4] = '\0';
     }

test "marvel"

before
Sega - Saturn | 8/18 | chd

after
Sega - Saturn | 13*/18 | chd
*confirmed 5 missing from DB

[pkos]

added conditional to remove raw "." character for ps1

        if (string_is_equal_fast(&raw_game_id[8], ".", 1))
        {
           raw_game_id[8] = raw_game_id[9];
           raw_game_id[9] = raw_game_id[10];
        }

test "marvel"

before
Sony - PlayStation | 44/62 | chd

after
Sony - PlayStation | 61/62 | chd

[pkos]

Additional scanning tests conducted by volunteers can help promote bug fixes. Enable logs to check serials.

[kivutar]

I compiled it and tested it on my catalog, on OSX. Everything went well. I think we can merge.

[pkos]

Possible change "_" to "-" as I'm looking at this one in hex editor.
|| (string_is_equal_fast(raw_game_id, "SLPS-", 5))

[pkos]

I've found a more GC specific suffix match in the samples, which overrides -EUR. Looking into it.

DL-DOL-G6TP-FRA

[pkos]

SLPS

How about we add SLPS- and SLPS_ both to be safe.

[pkos]

This is the conditional catching the country code "P" which is usually in the dat file as -EUR, but in once case here it actually should be -FRA. No other known indicator can separate the P country code.

  case 'P': /** NYI: P can also be P-UKV, P-AUS **/
     strcat(game_id, "-EUR");
     return true;

[pkos]

It's ready.