Campaign finance scrapper for legislators#2195
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Mephistic
left a comment
There was a problem hiding this comment.
LGTM - this is fantastic! Love the detailed breakdown comments on the data wrangling - those are going to be so handy for maintaining this pipeline going forward.
I know we still have unresolved questions that we're waiting on the subject matter expert for answers, but I'm fine merging this and building on top of it if anything radically changes based on that feedback.
|
|
||
| const OCPF_BASE_URL = "https://ocpf2.blob.core.windows.net/downloads/data2" | ||
|
|
||
| const YEARS = ["2025", "2026"] |
There was a problem hiding this comment.
If we want to start running this nightly, should we tie this to the years associated with the currentGeneralCourt for the next legislative session (or just use the current year and previous year)?
There was a problem hiding this comment.
Good call. We should tie it to the years associated with the currentGeneralCourt, which is the same as the election cycle for the next election (which is what the campaign fundraising is for). We can't use current year and previous year, since if it's currently an odd year, we only want that year, since the previous year will have been in a different legislative cycle.
Summary
This populates the finance tab on the legislator profile pages. See Issue #2118
It includes:
<Member_ID>
Checklist
firestore.indexes.json(Please do not only create indexes through the Firebase Web UI, even though the error messages may reccommend it - indexes created this way may be obliterated by subsequent deploys)Screenshots
Known issues
Steps to test/reproduce
For reproducing locally until firebase functions deployed:
Load members in firestore (run a few times with delay)
curl "http://localhost:5001/demo-dtp/us-central1/triggerPubsubFunction?scheduled=startMemberBatches"
Match member IDs:
curl -X POST http://localhost:5001/demo-dtp/us-central1/matchOcpfMembers
Run finance scraper:
curl -X POST http://localhost:5001/demo-dtp/us-central1/scrapeOcpfFinance
View Legislator Page:
Ex: http://localhost:3000/legislators/194/RLR0