Skip to content

Commit

Permalink
Added aliasMap.csv support to more scrapers
Browse files Browse the repository at this point in the history 
Updated `aliasMap.csv` comments

platform psp: added *.chd

Dev Container support w/Dockerfile (Gemba#23)

Co-authored-by: retrobit <6226450+retrobit@users.noreply.github.com>

Rel 3.10.0 (Gemba#26)

* simplified bash completion with built-in/provided functions...

plus:
- qmake: user defined install path
- doc: minor enhancements

* Prevention of very unlikely arithmetic overflow: Multiplication result converted to larger type

Rel 3.10.0 (Gemba#27)

* simplified bash completion with built-in/provided functions...

plus:
- qmake: user defined install path
- doc: minor enhancements

* Prevention of very unlikely arithmetic overflow: Multiplication result converted to larger type

* Version 3.10.0

Fix issue 28 (Gemba#32)

* simplified code with predefined function

* Fix Gemba#28

Fix cont. integration workflo (Gemba#33)

* update package index before install

Being pedantic/consistent

Pedantic/clarifying/consistent

Removed file used for testing

Reverted ES Game List scraper - doesn't make sense to use `aliasMap.csv` with this scraper

Not sure why this keeps getting added... I avoid staging it purposefully. Removed.

Scummvm: Changes to generate a ROM folder.svm/ as <game/> launcher instead of folder. (Gemba#35)

* Scummvm: Changes to generate a ROM `folder.svm/` as `<game/>` launcher instead of folder.

update README

Update from RetroPie Project (2024-02-22)

cf. RetroPie/EmulationStation#866

Cpp Std 17

Add new platforms (Gemba#37)

Updated platforms_idmap.csv and peas.json

Update docs:

- Removed passages about creating a symlink on RetroPie (now part of the scriptmodule).
- Edit paragraph about bash_completion on RetroPie

Fix Gemba#38 (Gemba#43)

Plus cornercase solved: When default ES inputFolder is symlinked from
/home/pi/RetroPie/roms to /home/pi/roms f.i. and user provides from that symlink
downwards rom files on CLI.

Version 3.10.2
  • Loading branch information
@retrobit
retrobit committed Mar 4, 2024
1 parent ec86792 commit 27178e5
Show file tree
Hide file tree
Showing 30 changed files with 4,999 additions and 3,317 deletions.
151 changes: 31 additions & 120 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,30 @@
# 🏢Skyscraper Enhanced+Reloaded
[![Tests](https://github.com/Gemba/skyscraper/actions/workflows/ci.yml/badge.svg)](https://github.com/Gemba/skyscraper/actions/workflows/ci.yml)
<h1 align="center">
<img src="docs/resources/skyscraper_banner.png" alt="Skyscraper" width="640px">
<br>
</h1>

<h4 align="center">Powerful and versatile game data scraper written in Qt and C++.</h4>

<p align="center">
<a href="https://github.com/Gemba/skyscraper/actions/workflows/ci.yml">
<img src="https://img.shields.io/github/actions/workflow/status/Gemba/skyscraper/ci.yml" alt="Build status">
</a>
<a href="https://github.com/Gemba/skyscraper/releases">
<img src="https://img.shields.io/github/v/release/Gemba/skyscraper.svg" alt="Releases">
</a>
<a href="https://github.com/Gemba/skyscraper/issues">
<img src="https://img.shields.io/badge/contributions-welcome-brightgreen.svg" alt="Issues">
</a>
</p>

<p align="center">
<a href="#how-to-install-skyscraper">Installation</a> •
<a href="#how-to-use-skyscraper">Quick Usage</a> •
<a href="https://gemba.github.io/skyscraper/">User Manual</a> •
<a href="https://gemba.github.io/skyscraper/CHANGELOG/">Changelog</a>
</p>

---

## Preface

Expand Down Expand Up @@ -68,7 +93,7 @@ $ brew link qt@5 --force
If that went well, proceed to the default installation instructions in the next section. It should work and give you a working installation of Skyscraper.

### Download, compile and install
When you've installed the prerequisites as described above, you can install Skyscraper by typing in the following commands:
When you've installed the prerequisites as described above for Linux or macOS, you can install Skyscraper by typing in the following commands:
```
$ cd
$ mkdir skysource
Expand Down Expand Up @@ -175,120 +200,6 @@ This feature is achieved by adding new config files:

These files are copied in the folder `/home/pi/.skyscraper` on RetroPie (or `/usr/local/etc/skyscraper/` in general) at the first run of the program if you want to edit them after an installation.

## Release notes

#### Version x.x.x (Features under consideration, all unimplemented)
* Add the option to scrape from cached data purely originating from certain scraping modules
* Consider making aliasMap the global baseName instead of just the search term base name. This will fix missing brackets in alias names being ignored later on
* Make all artwork types custom meaning that their type can be whatever the user wants. The sources will create ones with known types such as 'screenshot' and 'cover', but the user can import other types they define themselves, such as 'cabinet' or whatever else. Internally artwork is then handled by a list of artwork instead of 4 hardcoded types in GameEntry
* Introduce the ability to use <BASENAME> in the 'file="somefolder/<BASENAME>.png"' in artwork.xml which would then look for a file in 'resources' with the game basename.png allowing users to use their own custom artwork data as a workaround to adding new types to import (Thank you to 'jueank' for suggesting this)
* Add the option of scraping custom platforms by configuring them in the config with an alias to an already existing platform. Example: scrape 'pcenginecd' could be scraped as 'pcengine' in case you have those files in a 'roms/pcenginecd' folder instead of the pcengine folder. Check here: https://github.com/muldjord/skyscraper/issues/136
* Create a testmode for the artwork compositor that let's you quickly render an example to see if you got everything set up right in the artwork xml
* Allow 'region' to be a list similar to 'regionPrios'. When using 'region' it should simply keep the default priority list and add those from 'region' to the top. 'regionPrios' should still overwrite it entirely. Naming change probably a good idea, for instance rename 'region' to 'regionsPrefer' or something. 'regionPrios' should probably also be changed to 'regionsOverride'. (Thank you to 'corezon' for suggesting this).
* Implement a less rigid filename match for the 'import' module to allow for close match filenames
* Add support for grouping multi-disk games so they only have one entry in the gamelists. See issues/232 (Thank you to 'igno2k' for suggesting this)

#### Version 3.7.7-2
* Added script to quickly generate the mapMap.csv
* Added script to quickly generate the platforms.json from an existing platform.cpp file
* Moved the list of platforms and thier associated scrapers, file masks, and aliases to be loaded from a JSON file instead of hardcoded in CPP from [torresflo/skyscraper-enhanced](https://github.com/torresflo/skyscraper-enhanced)
* Added 'texture' image type to be stored in textures dir from [InfiRandia/skyscraper](https://github.com/InfiRandia/skyscraper)
* Updated MAME map to 0.245
* Added 'actionmax', 'naomi2', 'ps3', 'ps4', 'ps5', and 'xbox' platforms
* Added '*.chd' to 'naomi' and 'naomi2'
* Added '*.nca' to 'switch'
* Fixed the 'pc88' and 'x1' ScreenScraper IDs
* Added automatic generation of compiled builds on new tag

#### Version 3.7.7 (28th June 2022)
* Added '*.m3u' to 'segacd' platform (Thank you to user 'bmn001' for suggesting it)

#### Version 3.7.6 (12th June 2022)
* Added 'pico8' platform (Thank you to user 'zerojay' for suggesting it)

#### Version 3.7.5 (31st January 2022)
* Now allows ':' in Pegasus command

#### Version 3.7.4 (23rd January 2022)
* '--includefrom' and '--excludefrom' now works with relative paths (Thank you to use 'sleve_mcdichael' for reporting this
* Skipped file entries now conforms to same format as report files to allow using them with '--excludefrom' and '--includefrom' (Thank you to user 'TomFury' for suggesting this)

#### Version 3.7.3 (23rd January 2022)
* Added 'switch' platform (Thank you to user 'Redemp' for providing info)

#### Version 3.7.2 (20th January 2022)
* Skyscraper now only ignore files when using the '.skyscraperignore[tree]' files when scraping for new data. They will NOT be ignored when generating game lists

#### Version 3.7.1 (19th January 2022)
* Skyscraper will now ignore an entire tree of subfolders where a file called '.skyscraperignoretree' is found (Thank you to user 'sromeroi' for suggesting this)

#### Version 3.7.0 (17th January 2022)
* Moved '--fromfile' option to '--includefrom'. '--fromfile' still works, but is considered deprecated
* Moved '--includefiles' option to '--includepattern'. '--includefiles' still works, but is considered deprecated
* Moved '--excludefiles' option to '--excludepattern'. '--excludefiles' still works, but is considered deprecated
* Added '--excludefrom' option similar to '--includefrom' only the opposite (Thank you to user 'TomFury' for suggesting this)
* Skyscraper will now ignore any subfolders within the input folder where a file called '.skyscraperignore' is found (Thank you to user 'sromeroi' for suggesting this)
* Added platform 'easyrpg', only usable using the 'screenscraper' scraping module (Thank you to user 'zerojay' for suggesting this)

#### Version 3.6.16 (9th November 2021)
* Added platform 'moto' (Thank you to user 'Yserra' for suggesting it)

#### Version 3.6.15 (25th August 2021)
* Added 'chd' extension to 'atomiswave' platform (Thank you to user 'smeegoan' for reporting this)
* Fixed bug that caused 'T000000' to be added multiple times when skipping entries in ES gamelists (Thank you to user 'sleve_mcdichael' for reporting this)

#### Version 3.6.14 (5th August 2021)
* Added 'windows 3.x' as alias to Mobygames scraping module (Thank you to user 'ecto' for reporting this)

#### Version 3.6.13 (2nd June 2021)
* Added 'mediaFolderHidden' EmulationStation specific config option that will set the media folder to 'PLATFORM/.media' when set to true. This can speed up EmulationStation initial loading when using slow storage such as a network file system (Thank you to user 'XenuIsWatching' for suggesting this)

#### Version 3.6.12 (15th May 2021)
* Removed 'Simple Mode' as it was broken and deprecated. Use RetroPie script instead
* Removed A LOT of deprecated (and hidden) CLI options

#### Version 3.6.11 (6th May 2021)
* '--fromfile' now accepts both relative and absolute path to filename (Thank you to user 'sleve_mcdichael' for reporting this)
* Added platforms 'atarijaguarcd', 'pcenginecd' and 'channelf' (Thank you to user 'XenuIsWatching' for providing info)

#### Version 3.6.10 (12th April 2021)
* Added 'naomi' as platform
* Added 'msx2' as platform
* Added 'atomiswave' as platform

#### Version 3.6.9 (1st April 2021)
* Probably fixed `--flags unpack` which broke with 3.6.8 due to deprecated Qt function calls
* Probably fixed video conversion which broke with 3.6.8 due to deprecated Qt function calls

#### Version 3.6.8 (31st March 2021)
* Added 'cacheRefresh' config.ini option and added it to default config.ini for 'esgamelist' module (Thank you to user 'penx' for suggesting this)
* 'ScreenScraper APIv2 returned invalid / empty Json' message now no longer includes message to attach the error json file unless it's more than 64 bytes long
* Added 'ca' to region priority list (Thank you to user 'joaoluizcarvalho' for pointing this out)
* Added 'de', 'fr' and 'es' to default language priority list
* Added 'players' resource to '--cache report:missing' CLI option (Thank you to user 'peligwe' for reporting this)
* Added 'theInFront' config.ini option and 'theinfront' CLI flag (Thank you to user 'sleve_mcdichael' for suggesting this)
* 'unattend' and 'unattendSkip' can now be used in a scraping module section in config.ini (Thank you to user 'sleve_mcdichael' for reporting this)
* Added Qt version pre-processor checks for deprecated Qt methods to avoid compilation warning on newer Qt installations

#### Version 3.6.7 (27th January 2021)
* Added '*.rvz' extension for 'wii' and 'gc' platforms (Thank you to user 'tcamargo' for reporting this)
* Added '*.chd' extension for '3do' platform (Thank you to user 'Roudaku' for reporting this)
* Added 'openbor' platform. Exists in RetroPie as experimental platform (as requested by user 'hahnmt')
* 'neogeo' platform now uses 'box-2D' as cover instead of 'flyer' when scraping with the ScreenScraper module (Thank you to user 'retro81' for suggesting this)

#### Version 3.6.6 (21st December 2020)
* The 'esgamelist' module now looks for the 'gamelist.xml' file in the path set with '-g' instead of the path set with '-i' (Thank you to user 'c0d3h4x0r' for reporting this and helping me debug it)

#### Version 3.6.5 (18th December 2020)
* Now uses one shared networking instance, instead of one per thread
* 'daphne' platform entries now added as '<game>' entries instead of '<folder>' entries (Thank you to user 'clyde' for helping figure this out)
* Added 'nocropblack' cli flag and 'cropBlack="BOOL"' config.ini option that allows user to disable cropping of black borders around screenshots when compositing the final gamelist artwork (Thank you to user 'WindyWinston' for suggesting this)

#### Version 3.6.1 (27th October 2020)
* Added 'amigacd32' as a separate platform just because (Thank you to user 'HoraceAndTheSpider' for resetting The Matrix)

#### Version 3.6.0 (25th October 2020)
* Changed 'SecsSinceEpich' to 'MSecsSinceEpich' to support pre-5.8 Qt versions (Thank you to user 'Hazza4569' for reporting this)

#### Older releases
Release notes for older releases can be found [here](OLDERRELEASES.md).
## Previous Release Notes

Release notes for older releases which this fork builds on can be found [here](docs/OLDERRELEASES.md).
2 changes: 1 addition & 1 deletion VERSION
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
VERSION="3.10.0"
VERSION="3.10.2"
12 changes: 6 additions & 6 deletions aliasMap.csv
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@
#
# Note 1! This file is global and will be used when scraping ANY platform.
#
# Note 2! All bracket info in these titles are ignored. This is a search name alias
# NOT a title alias. It merely provides a different base name for Skyscraper
# when it creates the search query. It is meant to help scrape files that have
# abstract file names where the search based scraping modules have a hard time
# finding good results.
# Note 2! All bracket info in the search name alias is ignored.
# It merely provides a different base name for Skyscraper when it creates
# the search query. It is meant to help scrape files that have abstract filenames
# where the search-based scraping modules have a hard time finding good
# results, or you know the specific entry in direct-match scraping modules.
# If you wish to keep or add brackets to a title you can still do so by adding
# them to the actual file instead.
# them to the actual filename instead.
#
# Format (without the quotes):
# "Rom Filename (Europe);Use This Name Instead"
Expand Down
8 changes: 2 additions & 6 deletions docs/CLIHELP.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,6 @@ Also consider that almost all of these options are set at a useful default (and

Most of the options can also be set in the `/home/<USER>/.skyscraper/config.ini` file thus removing the need to type them on command line all the time. Check the config.ini doc [here](CONFIGINI.md) for more info on this.

!!! tip "For RetroPie users"

If you've installed Skyscraper through the RetroPie-Setup script, it is recommended to create a symbolic link to the executable. Do this by running `sudo ln -s /opt/retropie/supplementary/skyscraper/Skyscraper /usr/local/bin/Skyscraper`. This will allow you to just type `Skyscraper` when running it from command line.

### Programmable Completion

From version 3.9.3 onwards Skyscraper provides Bash ++tab++ completion (aka
Expand All @@ -18,9 +14,9 @@ programmable completion).
On RetroPie the
[scriptmodule](https://github.com/RetroPie/RetroPie-Setup/blob/master/scriptmodules/supplementary/skyscraper.sh)
will handle the installation. For the curious: It lands in
`~/.bash_completion.d/`.
`/etc/bash_completion.d/`.

On non-RetroPie Linux put the file
On non RaspiOS-based RetroPie-Installments put the file
`supplementary/bash-completion/Skyscraper.bash` into
`$XDG_DATA_HOME/bash-completion/completions/` respective to
`$HOME/.local/share/bash-completion/completions/`.
Expand Down
7 changes: 0 additions & 7 deletions docs/FAQ.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,6 @@ And now for the questions (with answers)...

Did you remember to quit EmulationStation before running Skyscraper? EmulationStation writes its current gamelists back to disk when it is closed. If you run Skyscraper while EmulationStation is also running, it will overwrite any gamelists you just generated. You need to quit EmulationStation before generating the gamelists with Skyscraper to avoid this. You can quit EmulationStation by pressing F4 on a connected keyboard, or by selecting quit from the menus.

??? Question "I installed Skyscraper from the RetroPie optional packages and want to run it from the terminal. But when I type `Skyscraper` it says `Skyscraper: command not found`. Why is that?"

The RetroPie-Setup script installs the Skyscraper executable in a non-default location that is not searched by the OS. You can fix this by creating a symbolic link to the executable in a location that the OS is aware of by running `sudo ln -s /opt/retropie/supplementary/skyscraper/Skyscraper /usr/local/bin/Skyscraper`. You might be asked for your sudo password, for RetroPie default is `raspberry`. If that command was succesful you should now be able to run Skyscraper simply with `Skyscraper` followed by any options you need.
!!! warning

Some people tend to run Skyscraper with `sudo Skyscraper ...`. Do _not_ do this! `sudo` is _not_ needed for running Skyscraper and will mess up your file permissions.

??? Question "I want to set up different options for different platforms. It's really tiresome setting it all on command-line. Can this be done easier?"

YES! Absolutely. In addition to taking options from the command-line, Skyscraper also reads the `/home/<USER>/.skyscraper/config.ini` file, which is where you should set everything up. This file allows you to configure things both globally, per-platform, per-frontend and per-scraping module. Read more about the available options and option priorities [here](CONFIGINI.md).
Expand Down

0 comments on commit 27178e5

Please sign in to comment.