USE 86 - new source mitlibwebsite extract command

[ghukill]

Purpose and background context

This PR adds support for generating an extract/harvest command for the new source mitlibwebsite. Additionally, it adds mitlibwebsite to the timdex and use aliases.

As noted in first and larger git commit, the approach was to add this functionality by following established patterns in command generation and helpers. However, this addition of a new harvester is beginning to stress some of the if/else patterns in the lambda code.

Previously, we had OAI or GIS, and if/else statements were arguably okay. Now, with the addition of mitlibwebsite there is quite a bit of branching.

At this point, I feel as though the lambdas are ready for a refactoring. Inputs and outputs could remain identical, but the code organization and branching inside the lambda handlers and helpers could be improved at this point. Proposing to wait on this work, but noting here in this PR.

How can a reviewer manually see the effects of these changes?

Before continuing, make sure to complete steps 1 and 2 from the README for local testing via AWS SAM.

Full Harvest

Run the following to invoke the lambda for a payload simulating a mitlibwebsite full extract:

sam local invoke -e tests/fixtures/event_payloads/mitlibwebsite-full-extract.json

The last line contains the lambda response, which formatted looks like:

{
  "run-date": "2025-10-14",
  "run-type": "full",
  "source": "mitlibwebsite",
  "verbose": true,
  "harvester-type": "browsertrix",
  "next-step": "transform",
  "extract": {
    "extract-command": [
      "--verbose",
      "harvest",
      "--config-yaml-file=s3://timdex-bucket/mitlibwebsite/config/mitlibwebsite.yaml",
      "--metadata-output-file=s3://timdex-bucket/mitlibwebsite/mitlibwebsite-2025-10-14-full-extracted-records-to-index.jsonl",
      "--sitemap=https://libraries.mit.edu/sitemap.xml",
      "--sitemap=https://libraries.mit.edu/news/sitemap.xml",
      "--sitemap-urls-output-file=s3://timdex-bucket/mitlibwebsite/last-sitemaps-urls.txt"
    ]
  }
}

Observe:

• No "previous" list of sitemap URLs is provided as input or output, because this "full" harvest will establish that file.

Daily Harvest

Run the following to invoke the lambda for a payload simulating a mitlibwebsite daily extract:

sam local invoke -e tests/fixtures/event_payloads/mitlibwebsite-daily-extract.json

The last line contains the lambda response, which formatted looks like:

{
  "run-date": "2025-10-14",
  "run-type": "daily",
  "source": "mitlibwebsite",
  "verbose": true,
  "harvester-type": "browsertrix",
  "next-step": "transform",
  "extract": {
    "extract-command": [
      "--verbose",
      "harvest",
      "--config-yaml-file=s3://timdex-bucket/mitlibwebsite/config/mitlibwebsite.yaml",
      "--metadata-output-file=s3://timdex-bucket/mitlibwebsite/mitlibwebsite-2025-10-14-daily-extracted-records-to-index.jsonl",
      "--sitemap=https://libraries.mit.edu/sitemap.xml",
      "--sitemap=https://libraries.mit.edu/news/sitemap.xml",
      "--sitemap-from-date=2025-10-13",
      "--sitemap-urls-output-file=s3://timdex-bucket/mitlibwebsite/last-sitemaps-urls.txt",
      "--previous-sitemap-urls-file=s3://timdex-bucket/mitlibwebsite/last-sitemaps-urls.txt"
    ]
  }
}

Observe:

• We do have previous sitemap URLs as both input and output now, where the daily harvest will analyze the previous run's sitemap URLs to find deletions.
• --sitemap-from-date is also added because of run-type="daily"

Includes new or updated dependencies?

NO

Changes expectations for external applications?

YES

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/USE-86

[coveralls]

Pull Request Test Coverage Report for Build 18536981192

Details

• 26 of 27 (96.3%) changed or added relevant lines in 4 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.06%) to 95.177%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
lambdas/helpers.py	10	11	90.91%

Totals
Change from base Build 18509110623:	0.06%
Covered Lines:	296
Relevant Lines:	311

---

💛 - Coveralls

[ehanson8]

Code looks good and SAM testing worked as expected!