Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Mobygames failure on manual search #834

Closed
mobedoor opened this issue May 2, 2024 · 3 comments · Fixed by #838
Closed

[Bug] Mobygames failure on manual search #834

mobedoor opened this issue May 2, 2024 · 3 comments · Fixed by #838
Assignees
@gantoine
Labels
bug Something isn't working

Comments

@mobedoor
Copy link

mobedoor commented May 2, 2024

RomM version
3.1.0

Describe the bug
Mobygames is not scraping games

To Reproduce
Steps to reproduce the behavior:

  1. Press manual search on a game
  2. Enter search query.
  3. No results
  4. Game does exist on mobygames.

Expected behavior
Scraped data about game

Desktop (please complete the following information):

  • OS: EndeavourOS
  • Browser Firefox
  • Version 125

Additional context
Additionally search by id throws the following error

romm     | INFO:	 [nginx][2024-05-02 19:35:36]	192.168.64.1 - - "GET /api/search/roms?rom_id=13&search_term=76920&search_by=ID&search_extended=false HTTP/1.1" 500 21 "http://localhost:8006/platform/5" "Mozilla/5.0 (X11; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0" rt=1.597 uct="0.000" uht="1.597" urt="1.597"
romm     | [2024-05-02 19:35:36 +0000] [52] [ERROR] Exception in ASGI application
romm     | Traceback (most recent call last):
romm     |   File "/backend/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
romm     |     result = await app(  # type: ignore[func-returns-value]
romm     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
romm     |   File "/backend/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
romm     |     return await self.app(scope, receive, send)
romm     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
romm     |   File "/backend/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
romm     |     await super().__call__(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
romm     |     await self.middleware_stack(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
romm     |     raise exc
romm     |   File "/backend/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
romm     |     await self.app(scope, receive, _send)
romm     |   File "/backend/handler/auth_handler/middleware.py", line 136, in __call__
romm     |     await self.app(scope, receive, send_wrapper)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/middleware/authentication.py", line 48, in __call__
romm     |     await self.app(scope, receive, send)
romm     |   File "/backend/handler/auth_handler/middleware.py", line 17, in __call__
romm     |     await super().__call__(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette_csrf/middleware.py", line 72, in __call__
romm     |     await self.app(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/middleware/cors.py", line 83, in __call__
romm     |     await self.app(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
romm     |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
romm     |     raise exc
romm     |   File "/backend/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
romm     |     await app(scope, receive, sender)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
romm     |     await self.middleware_stack(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
romm     |     await route.handle(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle
romm     |     await self.app(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/routing.py", line 79, in app
romm     |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
romm     |     raise exc
romm     |   File "/backend/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
romm     |     await app(scope, receive, sender)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
romm     |     response = await func(request)
romm     |                ^^^^^^^^^^^^^^^^^^^
romm     |   File "/backend/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
romm     |     raw_response = await run_endpoint_function(
romm     |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
romm     |   File "/backend/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
romm     |     return await dependant.call(**values)
romm     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
romm     |   File "/backend/lib/python3.11/site-packages/starlette/authentication.py", line 84, in async_wrapper
romm     |     return await func(*args, **kwargs)
romm     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
romm     |   File "/backend/endpoints/search.py", line 81, in search_rom
romm     |     merged_dict[item["name"]] = {
romm     |                 ~~~~^^^^^^^^
romm     | KeyError: 'name'
@mobedoor mobedoor added the bug Something isn't working label May 2, 2024
@gantoine
Copy link
Member

gantoine commented May 2, 2024

What search term are you using, and for which platform?

As for the error, seems Mobygames is returning a game with no title/name. I'll add a check for that in the next release.

@gantoine gantoine self-assigned this May 2, 2024
@gantoine gantoine changed the title [Bug] Mobygames not working [Bug] Mobygames failure on manual search May 2, 2024
@mobedoor
Copy link
Author

mobedoor commented May 3, 2024

For example, when searching for "Harry Potter and the Sorcerers stone" on Gameboy Color, I get

romm     |  - "GET /roms/15 HTTP/1.0" 200
romm     | INFO:	 [nginx][2024-05-03 03:38:52]	172.21.0.1 - - "GET /api/roms/15 HTTP/1.1" 200 1683 "http://localhost:8006/rom/15" "Mozilla/5.0 (X11; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0" rt=0.017 uct="0.000" uht="0.017" urt="0.017"
romm     |  - "GET /platforms/7 HTTP/1.0" 200
romm     | INFO:	 [nginx][2024-05-03 03:38:52]	172.21.0.1 - - "GET /api/platforms/7 HTTP/1.1" 200 133 "http://localhost:8006/rom/15" "Mozilla/5.0 (X11; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0" rt=0.013 uct="0.000" uht="0.013" urt="0.013"
romm     | INFO:	 [RomM][2024-05-03 03:38:56] 🔎 Searching metadata providers...
romm     | INFO:	 [RomM][2024-05-03 03:38:56] Searching by name: Harry Potter and the Sorcerer's Stone (USA, Europe) (En,Fr,De,Es,It,Nl,Pt,Sv,No,Da,Fi).zip
romm     | INFO:	 [RomM][2024-05-03 03:38:56] 🎮 gbc: Harry Potter and the Sorcerer's Stone (USA, Europe) (En,Fr,De,Es,It,Nl,Pt,Sv,No,Da,Fi).zip
romm     | INFO:	 [nginx][2024-05-03 03:38:56]	172.21.0.1 - - "GET /assets/moby.49a5e771.png HTTP/1.1" 200 24824 "http://localhost:8006/rom/15" "Mozilla/5.0 (X11; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0" rt=0.000 uct="-" uht="-" urt="-"
romm     | INFO:	 [RomM][2024-05-03 03:38:57] Results:
romm     |  - "GET /search/roms?rom_id=15&search_term=Harry+Potter+and+the+Sorcerer%27s+Stone+(USA,+Europe)+(En,Fr,De,Es,It,Nl,Pt,Sv,No,Da,Fi).zip&search_by=Name&search_extended=false HTTP/1.0" 200
romm     | INFO:	 [nginx][2024-05-03 03:38:57]	172.21.0.1 - - "GET /api/search/roms?rom_id=15&search_term=Harry+Potter+and+the+Sorcerer%27s+Stone+(USA,+Europe)+(En,Fr,De,Es,It,Nl,Pt,Sv,No,Da,Fi).zip&search_by=Name&search_extended=false HTTP/1.1" 200 2 "http://localhost:8006/rom/15" "Mozilla/5.0 (X11; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0" rt=1.482 uct="0.000" uht="1.481" urt="1.482"
romm     | INFO:	 [RomM][2024-05-03 03:38:59] 🔎 Searching metadata providers...
romm     | INFO:	 [RomM][2024-05-03 03:38:59] Searching by name: Harry Potter and the Sorcerer's Stone
romm     | INFO:	 [RomM][2024-05-03 03:38:59] 🎮 gbc: Harry Potter and the Sorcerer's Stone (USA, Europe) (En,Fr,De,Es,It,Nl,Pt,Sv,No,Da,Fi).zip
romm     | INFO:	 [RomM][2024-05-03 03:39:00] Results:
romm     |  - "GET /search/roms?rom_id=15&search_term=Harry+Potter+and+the+Sorcerer%27s+Stone&search_by=Name&search_extended=false HTTP/1.0" 200
romm     | INFO:	 [nginx][2024-05-03 03:39:00]	172.21.0.1 - - "GET /api/search/roms?rom_id=15&search_term=Harry+Potter+and+the+Sorcerer%27s+Stone&search_by=Name&search_extended=false HTTP/1.1" 200 2 "http://localhost:8006/rom/15" "Mozilla/5.0 (X11; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0" rt=1.182 uct="0.000" uht="1.182" urt="1.18

Upon searching by id (5416), I get

romm     | [2024-05-03 03:40:32 +0000] [46] [ERROR] Exception in ASGI application
romm     | Traceback (most recent call last):
romm     |   File "/backend/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
romm     |     result = await app(  # type: ignore[func-returns-value]
romm     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
romm     |   File "/backend/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
romm     |     return await self.app(scope, receive, send)
romm     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
romm     |   File "/backend/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
romm     |     await super().__call__(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
romm     |     await self.middleware_stack(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
romm     |     raise exc
romm     |   File "/backend/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
romm     |     await self.app(scope, receive, _send)
romm     |   File "/backend/handler/auth_handler/middleware.py", line 136, in __call__
romm     |     await self.app(scope, receive, send_wrapper)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/middleware/authentication.py", line 48, in __call__
romm     |     await self.app(scope, receive, send)
romm     |   File "/backend/handler/auth_handler/middleware.py", line 17, in __call__
romm     |     await super().__call__(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette_csrf/middleware.py", line 72, in __call__
romm     |     await self.app(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/middleware/cors.py", line 83, in __call__
romm     |     await self.app(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
romm     |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
romm     |     raise exc
romm     |   File "/backend/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
romm     |     await app(scope, receive, sender)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
romm     |     await self.middleware_stack(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
romm     |     await route.handle(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle
romm     |     await self.app(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/routing.py", line 79, in app
romm     |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
romm     |     raise exc
romm     |   File "/backend/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
romm     |     await app(scope, receive, sender)
romm     |   File "/backend/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
romm     |     response = await func(request)
romm     |                ^^^^^^^^^^^^^^^^^^^
romm     |   File "/backend/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
romm     |     raw_response = await run_endpoint_function(
romm     |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
romm     |   File "/backend/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
romm     |     return await dependant.call(**values)
romm     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
romm     |   File "/backend/lib/python3.11/site-packages/starlette/authentication.py", line 84, in async_wrapper
romm     |     return await func(*args, **kwargs)
romm     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
romm     |   File "/backend/endpoints/search.py", line 81, in search_rom
romm     |     merged_dict[item["name"]] = {
romm     |                 ~~~~^^^^^^^^
romm     | KeyError: 'name'

@Perrylicious Perrylicious mentioned this issue May 3, 2024
Closed
@gantoine
Copy link
Member

gantoine commented May 3, 2024

Harry Potter and the Sorcerers stone

that's likely because Sorcerer's has an apostrophe in it, and the mobygames API doesn't deal well (read: at all) with special characters)

Upon searching by id (5416), I get

this'll be fixed in the next release

@gantoine gantoine mentioned this issue May 3, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gantoine gantoine

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix serach by ID for mobygames rommapp/romm
2 participants
@gantoine @mobedoor