Scrape the new 6th Legislative Council pages instead #1
Conversation
There was a problem hiding this comment.
We've discovered that it's generally only sensible to have the scraper generate the terms table if the data in it is actually being scraped from that source. Otherwise, it's a lot better to just have the terms table in everypolitician-data maintained there. Switching this to a manually maintained file would avoid most of the work here, and be a lot easier to change in future.
|
Also, if the database needs to be deleted before run, this is a good opportunity to move the scraper on morph from Struan's account into the everypolitician-scrapers one (the only value in keeping it in an existing personal account is if there's history that would go missing on a fresh run). Then anyone in the team will be able to stop / start it, set the necessary morph vars, etc. |
|
OK, I've archived the terms table with just term 5 in this pull request: everypolitician/everypolitician-data#19158 |
There was an election in September and the 5th Legislative Council is over; the scraper also no longer worked because since then the pages for the 5th term have been changed to remove links to the people pages. This commit changes the scraper to scrape the 6th term pages instead.
mhl
force-pushed
the
update-to-add-6th-term
branch
from
54f16e9
to
49f1eaf
Compare
mhl
changed the title
|
@tmtmtmtm I've revised this - it's a much simpler change now. |
There has recently been an election in Hong Kong. We have archived the
data from the 5th Legislative Council, but now need to update the scraper
to fetch members from the new 6th Legislative Council instead.
(Note for the pull request, not in the commit mesage: the plan is to deploy
this under the everypolitician-scrapers account on morph.io.)