Skip to content

Commit

Permalink
comment out test to see if page already scraped
Browse files Browse the repository at this point in the history
Commenting this out for the first run as it isn't necessary and
throws an error because the db table to check  doesn't exist yet.
  • Loading branch information
equivalentideas committed May 2, 2015
1 parent cf25fc7 commit 8951f0e
Showing 1 changed file with 6 additions and 6 deletions.
12 changes: 6 additions & 6 deletions scraper.rb
Expand Up @@ -31,11 +31,11 @@ def save_media_release(page)
index = agent.get('http://www.police.nsw.gov.au/news/media_release_archives')

index.search('#content_div_111604 a').each do |link|
if !ScraperWiki.select("url from data where url='#{link.attr(:href)}'").empty?
puts "Skipping already saved media release #{link.text} #{link.attr(:href)}"
else
media_release_page = agent.get(link.attr(:href))
save_media_release(media_release_page)
end
# if !ScraperWiki.select("url from data where url='#{link.attr(:href)}'").empty?
# puts "Skipping already saved media release #{link.text} #{link.attr(:href)}"
# else
media_release_page = agent.get(link.attr(:href))
save_media_release(media_release_page)
# end
end

0 comments on commit 8951f0e

Please sign in to comment.