feat(db): Refactor dbRefresh and add dbUpdate

[bagelbits]

What does this do?

This PR significantly refactors and enhances the database update process, focusing on efficiency, maintainability, and reliability.

Key Changes & Improvements:

• Introduces Incremental Database Updates (dbUpdate.ts):

  • Adds a new dbUpdate.ts script that analyzes Git history (HEAD~1..HEAD) to find changed JSON files (src/**/*.json).
  • Processes only the changed files, performing targeted upsert or delete operations on individual records within the corresponding MongoDB collections.
  • Handles file additions, deletions, and renames correctly, including necessary updates to the associated index collection (e.g., 2014-collections).

• Refactors Shared Logic & Improves dbRefresh.ts:

  • Creates a central dbUtils.ts file to house shared utility functions (e.g., deriving collection/index names from file paths), reducing duplication (DRY).
  • Refactors the existing dbRefresh.ts script to:
    • Utilize the shared functions from dbUtils.ts for consistency.
    • Interact with MongoDB directly using the official mongodb driver instead of shelling out to mongoimport.

• Optimizes CI/CD Pipeline (ci.yml):

  • The deploy job now runs the new dbUpdate.ts script instead of the full dbRefresh.ts, resulting in significantly faster deployments that only apply necessary changes.
  • Adds a new manually triggerable workflow (manual-db-refresh.yml) as a "break glass" option to run the full dbRefresh.ts if needed.
  • General CI cleanup: uses npm ci for faster/more reliable installs and enables npm caching.

• Standardizes Environment & Configuration:

  • Updates workflows, package.json, and Dockerfile to consistently use Node.js v22.x (LTS).
  • Adds a .nvmrc file to enforce the Node version locally.
  • Modernizes the tsconfig.json configuration.

How was it tested?

1. Spun up docker-compose of API and DB
2. Checked both Acolyte and Aboleth
   1. Noted updated_at field for both
3. Ran this command and made sure it did nothing:

export MONGODB_URI=mongodb://localhost:27017/5e-database
npm run db:update

Checking for changed JSON files with status in the last commit...
No relevant JSON files changed in the last commit. No updates needed.

4. Changed armor_class value in Acolyte and made a git commit
5. Ran this command again:

export MONGODB_URI=mongodb://localhost:27017/5e-database
npm run db:update

Checking for changed JSON files with status in the last commit...
Found 1 changed JSON files:
 - Status: M, Path: src/2014/5e-SRD-Monsters.json

Connected successfully to MongoDB server

Processing src/2014/5e-SRD-Monsters.json (Status: M) for collection '2014-monsters'...
Attempting 1 upserts and 0 deletions in collection '2014-monsters'...
Bulk write completed: 0 upserted, 1 modified, 0 deleted.

Database update process finished successfully.
Database connection closed.

6. Verified that Acolyte updated in the API and the updated_at timestamp was changed for only Acolyte

Is there a Github issue this is resolving?

No. But I've been wanting to do this for so long.

Here's a fun image for your troubles

[Copilot]

Copilot reviewed 9 out of 13 changed files in this pull request and generated 1 comment.

Files not reviewed (4)

• .nvmrc: Language not supported
• Dockerfile: Language not supported
• package.json: Language not supported
• scripts/tsconfig.json: Language not supported

[Copilot]

Copilot reviewed 10 out of 14 changed files in this pull request and generated no comments.

Files not reviewed (4)

• .nvmrc: Language not supported
• Dockerfile: Language not supported
• package.json: Language not supported
• scripts/tsconfig.json: Language not supported

Comments suppressed due to low confidence (2)

scripts/update/gitUtils.ts:24

• This code assumes that the status code is always a single character. If Git returns a multi-character status in some cases, this could lead to incorrect status parsing. Consider using the entire status string or validating the expected format.

const status = parts[0][0] as ChangedFile['status'];

scripts/dbRefresh.ts:57

• Although the code later checks if the parsed data is an array, adding explicit validation immediately after parsing the JSON could help catch unexpected data formats earlier and improve error clarity.

data = JSON.parse(readFileSync(filepath, 'utf8'));

[github-actions]

🎉 This PR is included in version 3.13.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀