Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Akin to openstates/openstates#2681 this is the master ticket for 2019 updates
Please reference this issue in any PRs
Remaining States as of 1/16:
NV is throwing 500 errors on the endpoints that were previously used. The website seems to display the latest reps though, so the scraper might need an update. I'm going to try to find a first state that works before I dive into editing a scraper.
WA seems like it runs using
Going through the scraper contributing guide, it says after this stuff "And then the actual data scraping begins" with logged GET requests and whatnot. It doesn't seem like it's doing that at all. Any idea where to start to debug this?
also, possibly of interest to you guys (& anyone else looking to take a crack at this) I added a few experimental command line flags to the merge script
these are all fairly rough right now, but Colorado was a lot faster when I ran with
it got me about 85% of the way there without a ton of manual file moving/etc.
Actually more questions here first: I ran the merge helper and have some assumptions about what to do next:
For people who are the same (like this with a 0.70 score), update the file in the data directory.
For people who are not the same (like this with a 0.10 score), move the unused file to
The new IDs are because they're just GUIDs, there's no central database of legislator IDs like OCD has for Divisions/etc. So when you run the scraper, it creates a new GUID since it doesn't know any better. The old ID should be kept, since it'll match up with old data.
For people who are not the same, you're likely correct, you'd just want to retire the old legislator and bring in the new one.
I've created a simple script that can automate a lot of the process, it's not as complex and smart as merge.py, but it largely can perform all of the tasks as long as you check over it at the end/update the end/start dates: https://gist.github.com/csnardi/518cf39c0d0e909132f8ddec6f3817e9.