Merge pull request #780 from rommapp/master

v3.1-rc1

.gitignore

@@ -58,4 +58,3 @@ frontend/dev-dist
 # outside data
 switch_titledb.json
 switch_product_ids.json
-mame.xml

DEVELOPER_SETUP.md

@@ -86,8 +86,8 @@ npm install
 ### - Create symlink to library and resources
 ```sh
 mkdir assets/romm
-ln -s ../../../romm_mock/resources assets/romm/resources
-ln -s ../../../romm_mock/assets assets/romm/assets
+ln -s ../backend/romm_mock/resources assets/romm/resources
+ln -s ../backend/romm_mock/assets assets/romm/assets
 ```
 
 ### - Run the frontend

README.md

@@ -18,21 +18,20 @@
   </div>
 </div>
 
-# ⚠️ Breaking changes in version 3.0 ⚠️
-
-Version 3.0 introduces exciting new fetures that require some changes to how RomM is setup and configured. **If you're currently running a 2.x version, please review the [migration guide](https://github.com/rommapp/romm/wiki/Upgrading-to-3.0) before upgrading.**
+> [!WARNING]  
+> Version 3.0 introduces exciting new fetures that require some changes to how RomM is setup and configured. **If you're currently running a 2.x version, please review the [migration guide](https://github.com/rommapp/romm/wiki/Upgrading-to-3.0) before upgrading.**
 
 # Overview
 
 RomM (ROM Manager) allows you to scan, enrich, and browse your game collection with a clean and responsive interface. With support for multiple platforms, various naming schemes and custom tags, RomM is a must-have for anyone who plays on emulators.
 
 ## Features
 
-- Scans your existing games library and enchances it with metadata from [IGDB][igdb]
+- Scans your existing games library and enchances it with metadata from [IGDB][igdb] and [MobyGames][mobygames]
 - Supports a large number of **[platforms][platform-support]**
 - Play games directly from the browser using [EmulatorJS][wiki-emulatorjs-url]
 - Share your library with friend while [limiting access and permissions][authentication]
-- Supports [MAME][mame-xml-update], [Nintendo Switch][switch-titledb-update] and PS2 naming schemes
+- Supports MAME, Nintendo Switch, and Sony Playstation naming schemes
 - Detects and groups **multifile games** (e.g. PS1 games with multiple CDs)
 - Can [parse tags][tag-support] in filenames (e.g. (E), (USA), (rev v1), etc.)
 - View, upload, update, and delete games from any modern web browser
@@ -47,14 +46,13 @@ RomM (ROM Manager) allows you to scan, enrich, and browse your game collection w
 
 Before running the [image][docker-tags], ensure that Docker is installed and running on your system.
 
-1. Generate an API key for [IGDB][igdb] and set the `IGDB_CLIENT_ID` and `IGDB_CLIENT_SECRET` variables. This step is essential for running a library scan. Instructions for generating the ID and Secret can be found [here][igdb-api]. Note that IGDB requires a Twitch account with 2FA enabled to generate the ID and Secret.
+1. [Generate API keys][wiki-generate-api-keys] for IGDB and/or MobyGames to fetch metadata.
 2. Verify that your library folder structure matches one of the options listed in the [folder structure][folder-structure] section.
-3. Create a docker-compose.yml file by referring to the example [docker-compose.yml][docker-compose-example] file for guidance, and customize it for your setup with [the available environment variables](wiki-env-variables).
+3. Create a docker-compose.yml file by referring to the example [docker-compose.yml][docker-compose-example] file for guidance, and customize it for your setup with [the available environment variables][wiki-env-variables].
 4. Launch the container(s) with `docker-compose up -d`.
 
-### Troubleshooting
-
-If you are having issues with RomM, please review the [wiki page][wiki-troubleshooting-url] for troubleshooting steps and common issues.
+> [!NOTE]  
+> **If you are having issues with RomM, please review the [wiki page][wiki-troubleshooting-url] for troubleshooting steps and common issues.**
 
 # Configuration
 
@@ -121,11 +119,12 @@ As mentioned in the installation section, RomM requires a specific folder struct
  </tr>
 </table>
 
-For folder naming conventions, review the [Platform Support][platform-support] section. To override default system names in the folder structure (if your directories are named differently), see the [Configuration File][configuration-file] section.
+> [!TIP]
+> For folder naming conventions, review the [Platform Support][platform-support] section. To override default system names in the folder structure (if your directories are named differently), see the [Configuration File][configuration-file] section.
 
 ## Configuration File
 
-RomM's "understanding" of your library can be configured with a `config.yaml` file. Anytime that file is change, **you must restart the container for changes to take effect.** Refer to the [example config.yml][configuration-file-example] file for guidance on how to configure it, and the [example docker-compose.yml][docker-compose-example] file on how to mount it into the container.
+RomM's "understanding" of your library can be configured with a `config.yaml` file or through the `config` tab in the `Control Panel` under `Settings` section. Refer to the [example config.yml][configuration-file-example] file for guidance on how to configure it, and the [example docker-compose.yml][docker-compose-example] file on how to mount it into the container.
 
 ## Scheduler
 
@@ -180,8 +179,6 @@ Here are a few projects that we think you might like:
 [platform-support]: #-platform-support
 [authentication]: #-authentication
 [tag-support]: #-tag-support
-[switch-titledb-update]: #-switch-titledb-update
-[mame-xml-update]: #-mame-xml-update
 [configuration-file]: #-configuration-file
 
 <!-- Files -->
@@ -199,6 +196,7 @@ Here are a few projects that we think you might like:
 [wiki-emulatorjs-url]: https://github.com/rommapp/romm/wiki/EmulatorJS-Player
 [wiki-env-variables]: https://github.com/rommapp/romm/wiki/Environment-Variables
 [wiki-scheduled-tasks]: https://github.com/rommapp/romm/wiki/Scheduled-Tasks
+[wiki-generate-api-keys]: https://github.com/rommapp/romm/wiki/Generate-API-Keys
 
 <!-- Badges -->
 
@@ -224,8 +222,8 @@ Here are a few projects that we think you might like:
 <!-- External links -->
 
 [docker-tags]: https://hub.docker.com/r/rommapp/romm/tags
-[igdb]: https://www.igdb.com/
-[igdb-api]: https://api-docs.igdb.com/#getting-started
+[igdb-api]: https://api-docs.igdb.com/#account-creation
+[mobygames-api]: https://www.mobygames.com/info/api/
 [titleid-program-id-url]: https://switchbrew.org/w/index.php?title=Title_list/Games&mobileaction=toggle_view_desktop
 [igdb-platforms-list]: https://www.igdb.com/platforms
 [big-bear-casaos]: https://github.com/bigbeartechworld/big-bear-casaos

backend/alembic/versions/0015_mobygames_data.py

@@ -0,0 +1,43 @@
+"""empty message
+
+Revision ID: 0015_mobygames_data
+Revises: 0014_asset_files
+Create Date: 2024-02-13 17:57:25.936825
+
+"""
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import mysql
+
+# revision identifiers, used by Alembic.
+revision = "0015_mobygames_data"
+down_revision = "0014_asset_files"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table("platforms", schema=None) as batch_op:
+        batch_op.add_column(sa.Column("moby_id", sa.Integer(), nullable=True))
+
+    with op.batch_alter_table("roms", schema=None) as batch_op:
+        batch_op.add_column(sa.Column("moby_id", sa.Integer(), nullable=True))
+        batch_op.add_column(sa.Column("moby_metadata", mysql.JSON(), nullable=True))
+
+    with op.batch_alter_table("roms", schema=None) as batch_op:
+        batch_op.execute("update roms set moby_metadata = '\\{\\}'")
+
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table("roms", schema=None) as batch_op:
+        batch_op.drop_column("moby_metadata")
+        batch_op.drop_column("moby_id")
+
+    with op.batch_alter_table("platforms", schema=None) as batch_op:
+        batch_op.drop_column("moby_id")
+
+    # ### end Alembic commands ###

backend/config/__init__.py

@@ -28,6 +28,7 @@
 # REDIS
 REDIS_HOST: Final = os.environ.get("REDIS_HOST", "127.0.0.1")
 REDIS_PORT: Final = os.environ.get("REDIS_PORT", 6379)
+REDIS_PASSWORD: Final = os.environ.get("REDIS_PASSWORD")
 
 # IGDB
 IGDB_CLIENT_ID: Final = os.environ.get(
@@ -40,6 +41,9 @@
 # STEAMGRIDDB
 STEAMGRIDDB_API_KEY: Final = os.environ.get("STEAMGRIDDB_API_KEY", "")
 
+# MOBYGAMES
+MOBYGAMES_API_KEY: Final = os.environ.get("MOBYGAMES_API_KEY", "")
+
 # DB DRIVERS
 ROMM_DB_DRIVER: Final = os.environ.get("ROMM_DB_DRIVER", "mariadb")
 
@@ -51,6 +55,9 @@
 )
 DISABLE_CSRF_PROTECTION = os.environ.get("DISABLE_CSRF_PROTECTION", "false") == "true"
 
+# SCANS
+SCAN_TIMEOUT: Final = int(os.environ.get("SCAN_TIMEOUT", 60 * 60 * 4))  # 4 hours
+
 # TASKS
 ENABLE_RESCAN_ON_FILESYSTEM_CHANGE: Final = (
     os.environ.get("ENABLE_RESCAN_ON_FILESYSTEM_CHANGE", "false") == "true"
@@ -72,10 +79,3 @@
     "SCHEDULED_UPDATE_SWITCH_TITLEDB_CRON",
     "0 4 * * *",  # At 4:00 AM every day
 )
-ENABLE_SCHEDULED_UPDATE_MAME_XML: Final = (
-    os.environ.get("ENABLE_SCHEDULED_UPDATE_MAME_XML", "false") == "true"
-)
-SCHEDULED_UPDATE_MAME_XML_CRON: Final = os.environ.get(
-    "SCHEDULED_UPDATE_MAME_XML_CRON",
-    "0 5 * * *",  # At 5:00 AM every day
-)

backend/config/config_manager.py

@@ -84,7 +84,7 @@ def get_db_engine() -> str:
         if ROMM_DB_DRIVER == "mariadb":
             if not DB_USER or not DB_PASSWD:
                 log.critical(
-                    "Missing database credentials. Please check your configuration file"
+                    "Missing database credentials, check your environment variables!"
                 )
                 sys.exit(3)
 

backend/endpoints/heartbeat.py

@@ -1,14 +1,14 @@
 from config import (
     ENABLE_RESCAN_ON_FILESYSTEM_CHANGE,
     ENABLE_SCHEDULED_RESCAN,
-    ENABLE_SCHEDULED_UPDATE_MAME_XML,
     ENABLE_SCHEDULED_UPDATE_SWITCH_TITLEDB,
     RESCAN_ON_FILESYSTEM_CHANGE_DELAY,
     SCHEDULED_RESCAN_CRON,
-    SCHEDULED_UPDATE_MAME_XML_CRON,
     SCHEDULED_UPDATE_SWITCH_TITLEDB_CRON,
 )
 from endpoints.responses.heartbeat import HeartbeatResponse
+from handler.metadata_handler.igdb_handler import IGDB_API_ENABLED
+from handler.metadata_handler.moby_handler import MOBY_API_ENABLED
 from fastapi import APIRouter
 from handler import github_handler
 
@@ -26,6 +26,11 @@ def heartbeat() -> HeartbeatResponse:
     return {
         "VERSION": github_handler.get_version(),
         "NEW_VERSION": github_handler.check_new_version(),
+        "ANY_SOURCE_ENABLED": IGDB_API_ENABLED or MOBY_API_ENABLED,
+        "METADATA_SOURCES": {
+            "IGDB_API_ENABLED": IGDB_API_ENABLED,
+            "MOBY_API_ENABLED": MOBY_API_ENABLED,
+        },
         "WATCHER": {
             "ENABLED": ENABLE_RESCAN_ON_FILESYSTEM_CHANGE,
             "TITLE": "Rescan on filesystem change",
@@ -44,11 +49,5 @@ def heartbeat() -> HeartbeatResponse:
                 "TITLE": "Scheduled Switch TitleDB update",
                 "MESSAGE": "Updates the Nintedo Switch TitleDB file",
             },
-            "MAME_XML": {
-                "ENABLED": ENABLE_SCHEDULED_UPDATE_MAME_XML,
-                "CRON": SCHEDULED_UPDATE_MAME_XML_CRON,
-                "TITLE": "Scheduled MAME XML update",
-                "MESSAGE": "Updates the MAME XML file",
-            },
         }
     }

backend/endpoints/responses/heartbeat.py

@@ -14,11 +14,17 @@ class TaskDict(WatcherDict):
 class SchedulerDict(TypedDict):
     RESCAN: TaskDict
     SWITCH_TITLEDB: TaskDict
-    MAME_XML: TaskDict
+
+
+class MetadataSourcesDict(TypedDict):
+    IGDB_API_ENABLED: bool
+    MOBY_API_ENABLED: bool
 
 
 class HeartbeatResponse(TypedDict):
     VERSION: str
     NEW_VERSION: str
     WATCHER: WatcherDict
     SCHEDULER: SchedulerDict
+    ANY_SOURCE_ENABLED: bool
+    METADATA_SOURCES: MetadataSourcesDict

backend/endpoints/responses/platform.py

@@ -9,6 +9,7 @@ class PlatformSchema(BaseModel):
     fs_slug: str
     igdb_id: Optional[int] = None
     sgdb_id: Optional[int] = None
+    moby_id: Optional[int] = None
     name: Optional[str]
     logo_path: str
     rom_count: int

backend/endpoints/responses/rom.py

@@ -1,33 +1,36 @@
 import re
-from typing import Optional
+from typing import Optional, get_type_hints
+from typing_extensions import TypedDict, NotRequired
 
 from endpoints.responses.assets import SaveSchema, ScreenshotSchema, StateSchema
 from fastapi import Request
 from fastapi.responses import StreamingResponse
 from handler import socket_handler
-from handler.igdb_handler import IGDBRelatedGame
+from handler.metadata_handler.igdb_handler import IGDBMetadata
+from handler.metadata_handler.moby_handler import MobyMetadata
 from pydantic import BaseModel, computed_field, Field
 from models.rom import Rom
-from typing_extensions import TypedDict, NotRequired
 
 
 SORT_COMPARE_REGEX = r"^([Tt]he|[Aa]|[Aa]nd)\s"
 
-
-class RomMetadata(TypedDict):
-    expansions: NotRequired[list[IGDBRelatedGame]]
-    dlcs: NotRequired[list[IGDBRelatedGame]]
-    remasters: NotRequired[list[IGDBRelatedGame]]
-    remakes: NotRequired[list[IGDBRelatedGame]]
-    expanded_games: NotRequired[list[IGDBRelatedGame]]
-    ports: NotRequired[list[IGDBRelatedGame]]
-    similar_games: NotRequired[list[IGDBRelatedGame]]
+RomIGDBMetadata = TypedDict(
+    "RomIGDBMetadata",
+    {k: NotRequired[v] for k, v in get_type_hints(IGDBMetadata).items()},
+    total=False,
+)
+RomMobyMetadata = TypedDict(
+    "RomMobyMetadata",
+    {k: NotRequired[v] for k, v in get_type_hints(MobyMetadata).items()},
+    total=False,
+)
 
 
 class RomSchema(BaseModel):
     id: int
     igdb_id: Optional[int]
     sgdb_id: Optional[int]
+    moby_id: Optional[int]
 
     platform_id: int
     platform_slug: str
@@ -45,16 +48,15 @@ class RomSchema(BaseModel):
     summary: Optional[str]
 
     # Metadata fields
-    total_rating: Optional[str]
-    aggregated_rating: Optional[str]
     first_release_date: Optional[int]
     alternative_names: list[str]
     genres: list[str]
     franchises: list[str]
     collections: list[str]
     companies: list[str]
     game_modes: list[str]
-    igdb_metadata: Optional[RomMetadata]
+    igdb_metadata: Optional[RomIGDBMetadata]
+    moby_metadata: Optional[RomMobyMetadata]
 
     path_cover_s: Optional[str]
     path_cover_l: Optional[str]

backend/endpoints/responses/search.py

@@ -1,5 +1,10 @@
-from handler.igdb_handler import IGDBRom
+from pydantic import BaseModel
 
-
-class SearchRomSchema(IGDBRom):
-    pass
+class SearchRomSchema(BaseModel):
+    igdb_id: int | None = None
+    moby_id: int | None = None
+    slug: str
+    name: str
+    summary: str
+    url_cover: str
+    url_screenshots: list[str]