Skip to content

Commit

Permalink
Added safety count.
Browse files Browse the repository at this point in the history
  • Loading branch information
MichaelBone committed Aug 16, 2018
1 parent d9f844e commit 8fdb3a7
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions scraper.rb
Expand Up @@ -73,6 +73,7 @@
# p summary_page.title.strip

p "Paging"
count = 0
das_data = []
while summary_page
table = summary_page.root.at_css('.ContentPanel')
Expand All @@ -88,6 +89,11 @@
next_page_path = next_page_img['onclick'].split(',').find { |e| e =~ /.*PageNumber=\d+.*/ }.gsub('"', '').strip
p "Found another page: " + next_page_path
summary_page = agent.get "#{base_url}#{next_page_path}"
count = count + 1
if count > 2
p "Stopping paging after " + count + "pages."
break
end
end
end

Expand Down

0 comments on commit 8fdb3a7

Please sign in to comment.