DataScraper

Purpose

The purpose of the DataScraper is to build a database of mods from data retrieved the Nexus API. It does this by retrieving data from that API and then sending it to the internal API for processing and storage.

Setup

In order to use the scraper for collecting mod data, you need to follow the below steps.

1. Obtain one or more Nexusmods API keys from https://www.nexusmods.com/users/myaccount?tab=api. Use the field named Personal API Key. If you wish to use more than one API key to avoid ratelimiting delays, you must register multiple Nexus accounts. Each account requires a unique email address.

2. Duplicate the file settings.example.json from the repository root and rename it settings.json. This is where all your credentials will be stored (the filename settings.json has been added to .gitignore). Fill in the fields as follows (if you are only using one Nexus API key, leave it in the array anyway):

   {
       "api_key":["nexus API key one", 
           "nexus API key two", 
           "nexus API key three"],
       "auth_key": "internal API authentication key (ask Tom)",
       "api_url": "https://arch.tdpain.net/api/nexusmod/create/",
       "game": "skyrim",
       "range": [99142, 100250]
   }

   api_url should remain unchanged. When selecting a game to use, put the game domain name in the game field, and add the mod ID range into the range section, as shown in the example.

3. Run the scraper. After each run, you should check the error.log file the script creates to check if the script stopped prematurely, and if so, why.

Script rundown

scraper.py is the main script for the scraper. It has full error logging to error.log using Loguru. The script has a method to switch between API keys when only 5 requests are left remaining on a certain token, and if there are no tokens with more than 5 requests remaining on them, the script will wait until the next hour and for the ratelimit to reset.

The range of mods for the script to process is defined by the values entered in settings.json.

To gather mod information, the mod page on Nexusmods is first downloaded and parsed using BeautifulSoup in order to determine if it is hidden, missing or otherwise unavailable.

The script will then fire off a request to the Nexus API and retrieve data from there, and send everything it has collected about that one mod to the internal API. Once this is complete, it moves onto the next mod.