Repository navigation
v2.1.2
What changed
Coverage scraper redesign
Replaced Firecrawl search with Google News RSS for article discovery. Firecrawl search was hallucinating URLs — generating plausible-looking article links that pointed to nonexistent pages. 9 of the original 27 coverage articles (33%) were either fabricated or irrelevant.
New discovery pipeline:
- Google News RSS as primary article discovery (real, verified URLs)
- Existing RSS feeds for NJ Transit, NJ.com, Gothamist (unchanged)
- Firecrawl for content scraping only (excerpts, metadata)
New validation gates:
- HTTP HEAD request — every candidate URL must return 200
- Content relevance check — scraped text must mention Portal Bridge
- Link health checker (
--check-links) — audits all existing article URLs
About page
Added methodology page (about.html) explaining data sourcing, verification process, and testing approach. Translated into all 10 languages.
Article cleanup
Removed 9 broken/irrelevant articles:
- 3 TAPinto articles with fabricated URL paths
- 4 NYTimes articles returning 404
- 2 NJ Transit pages unrelated to Portal Bridge (parking rates, Presidents' Day schedule)
Coverage feed: 27 → 18 verified articles.
Full changelog: v2.1.1...v2.1.2