Skip to content

Campaign finance scrapper for legislators#2195

Merged
Mephistic merged 1 commit into
codeforboston:mainfrom
J-C-L:finance-scrapper
Jul 22, 2026
Merged

Campaign finance scrapper for legislators#2195
Mephistic merged 1 commit into
codeforboston:mainfrom
J-C-L:finance-scrapper

Conversation

@J-C-L

@J-C-L J-C-L commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

This populates the finance tab on the legislator profile pages. See Issue #2118
It includes:

  • Fetches campaign finance data from the OCPF website [currently this is a run on demand function. Once data is validated, we'll switch to nightly scheduled run
  • Stores campaign finance data per legislator in Firestore under new collection: generalCourts > 194 > membersFinance
    <Member_ID>
  • Displays campaign finance summary on finance tab of legislator page for each legislator

Checklist

  • [Y] On the frontend, I've made my strings translate-able.
  • [N/A ] If I've added shared components, I've added a storybook story.
  • [Y ] I've made pages responsive and look good on mobile.
  • [ N/A] If I've added new Firestore queries, I've added any new required indexes to firestore.indexes.json (Please do not only create indexes through the Firebase Web UI, even though the error messages may reccommend it - indexes created this way may be obliterated by subsequent deploys)

Screenshots

finance-tab-screenshot

Known issues

  • scrapeOcpfFinance function must be run after matchOcpfMembers, so court member IDs are already associated in firestore with OCPF IDs
  • Code is safe to push while legislator pages are behind a flag, and hidden in prod.
  • Still need to consult with campaign finance expert on some data choices made here.
  • "small donors" box is not yet populated. Need to decide what we want to use there (<$50 or <$200)

Steps to test/reproduce

For reproducing locally until firebase functions deployed:

  1. Load members in firestore (run a few times with delay)
    curl "http://localhost:5001/demo-dtp/us-central1/triggerPubsubFunction?scheduled=startMemberBatches"

  2. Match member IDs:
    curl -X POST http://localhost:5001/demo-dtp/us-central1/matchOcpfMembers

  3. Run finance scraper:
    curl -X POST http://localhost:5001/demo-dtp/us-central1/scrapeOcpfFinance

  4. View Legislator Page:
    Ex: http://localhost:3000/legislators/194/RLR0

@vercel

vercel Bot commented Jul 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
maple-dev Ready Ready Preview, Comment Jul 22, 2026 1:35am

Request Review

@Mephistic Mephistic left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - this is fantastic! Love the detailed breakdown comments on the data wrangling - those are going to be so handy for maintaining this pipeline going forward.

I know we still have unresolved questions that we're waiting on the subject matter expert for answers, but I'm fine merging this and building on top of it if anything radically changes based on that feedback.


const OCPF_BASE_URL = "https://ocpf2.blob.core.windows.net/downloads/data2"

const YEARS = ["2025", "2026"]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to start running this nightly, should we tie this to the years associated with the currentGeneralCourt for the next legislative session (or just use the current year and previous year)?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call. We should tie it to the years associated with the currentGeneralCourt, which is the same as the election cycle for the next election (which is what the campaign fundraising is for). We can't use current year and previous year, since if it's currently an odd year, we only want that year, since the previous year will have been in a different legislative cycle.

@J-C-L
J-C-L force-pushed the finance-scrapper branch from c903622 to 9977ecc Compare July 22, 2026 01:29
@Mephistic
Mephistic merged commit 91d239d into codeforboston:main Jul 22, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants