Skip to content

Commit

Permalink
Create an id for councillors
Browse files Browse the repository at this point in the history
This helps us distinguish between councillors with the same name when
using the csv_to_popolo converter:
https://github.com/equivalentideas/csv_to_popolo

This idea came from discussion on the issue here:
openaustralia/planningalerts#896 (comment)
  • Loading branch information
equivalentideas committed Feb 18, 2016
1 parent 4eab707 commit bf5dbde
Showing 1 changed file with 8 additions and 0 deletions.
8 changes: 8 additions & 0 deletions scraper.rb
Expand Up @@ -28,6 +28,11 @@ def simplify_name(text)
parts[first..last].join(" ")
end

def create_id(council, name)
components = council + "/" + name
components.downcase.gsub(" ","_")
end

def parse_council(text)
count = 0
name = text.lines[0].strip
Expand All @@ -48,6 +53,7 @@ def parse_council(text)
if (mayor_line[0] == "Mayor" or mayor_line[0] == "Lord Mayor") and not mayor_line[1].empty?
councillor_name = simplify_name(mayor_line[1].strip)
ScraperWiki.save_sqlite(["councillor", "council_name"], {
"id" => create_id(name, councillor_name),
"councillor" => councillor_name,
"position" => mayor_line[0],
"council_name" => name,
Expand All @@ -60,6 +66,7 @@ def parse_council(text)
if deputy_line[0] == "Deputy" and not deputy_line[1].empty?
councillor_name = simplify_name(deputy_line[1].strip)
ScraperWiki.save_sqlite(["councillor", "council_name"], {
"id" => create_id(name, councillor_name),
"councillor" => councillor_name,
"position" => "deputy mayor",
"council_name" => name,
Expand All @@ -71,6 +78,7 @@ def parse_council(text)
text.lines[area_line_no + 7].split(",").each do |t|
next if t.strip.empty?
ScraperWiki.save_sqlite(["councillor", "council_name"], {
"id" => create_id(name, simplify_name(t.strip)),
"councillor" => simplify_name(t.strip),
"council_name" => name,
"council_website" => website})
Expand Down

0 comments on commit bf5dbde

Please sign in to comment.