Skip to content

v2.1.2

Choose a tag to compare

@jamditis jamditis released this 14 Feb 06:35
· 141 commits to main since this release

What changed

Coverage scraper redesign

Replaced Firecrawl search with Google News RSS for article discovery. Firecrawl search was hallucinating URLs — generating plausible-looking article links that pointed to nonexistent pages. 9 of the original 27 coverage articles (33%) were either fabricated or irrelevant.

New discovery pipeline:

  • Google News RSS as primary article discovery (real, verified URLs)
  • Existing RSS feeds for NJ Transit, NJ.com, Gothamist (unchanged)
  • Firecrawl for content scraping only (excerpts, metadata)

New validation gates:

  • HTTP HEAD request — every candidate URL must return 200
  • Content relevance check — scraped text must mention Portal Bridge
  • Link health checker (--check-links) — audits all existing article URLs

About page

Added methodology page (about.html) explaining data sourcing, verification process, and testing approach. Translated into all 10 languages.

Article cleanup

Removed 9 broken/irrelevant articles:

  • 3 TAPinto articles with fabricated URL paths
  • 4 NYTimes articles returning 404
  • 2 NJ Transit pages unrelated to Portal Bridge (parking rates, Presidents' Day schedule)

Coverage feed: 27 → 18 verified articles.

Full changelog: v2.1.1...v2.1.2