Skip to content

Commit

Permalink
Modernise scraper
Browse files Browse the repository at this point in the history
  • Loading branch information
tmtmtmtm committed Jan 25, 2016
1 parent d6c90d2 commit 94fa40a
Showing 1 changed file with 3 additions and 11 deletions.
14 changes: 3 additions & 11 deletions scraper.rb
Original file line number Diff line number Diff line change
@@ -1,16 +1,8 @@
#!/bin/env ruby
# encoding: utf-8

require 'scraperwiki'
require 'wikidata/fetcher'
require 'rest-client'

@pages = [ 'Categoría:Diputados_de_la_X_Legislatura_de_España' ]

@pages.map { |c| WikiData::Category.new(c, 'es').wikidata_ids }.flatten.uniq.each do |id|
data = WikiData::Fetcher.new(id: id).data('es') or next
ScraperWiki.save_sqlite([:id], data)
end

warn RestClient.post ENV['MORPH_REBUILDER_URL'], {} if ENV['MORPH_REBUILDER_URL']

names = WikiData::Category.new('Categoría:Diputados_de_la_X_Legislatura_de_España', 'es').member_titles
EveryPolitician::Wikidata.scrape_wikidata(names: { es: names }, output: false)
warn EveryPolitician::Wikidata.notify_rebuilder

0 comments on commit 94fa40a

Please sign in to comment.