Skip to content

Commit

Permalink
Merge pull request #2 from tseyiokorodudu/master
Browse files Browse the repository at this point in the history
Added proxy support and Fixed scraper error
  • Loading branch information
mlandauer committed Dec 9, 2021
2 parents 06a19fe + c4beebf commit 881dbce
Showing 1 changed file with 9 additions and 1 deletion.
10 changes: 9 additions & 1 deletion scraper.rb
Expand Up @@ -16,11 +16,19 @@ def parse_date(s)
agent = Mechanize.new
page = agent.get(url)

if ENV["MORPH_AUSTRALIAN_PROXY"]
# On morph.io set the environment variable MORPH_AUSTRALIAN_PROXY to
# http://morph:password@au.proxy.oaf.org.au:8888 replacing password with
# the real password.
puts "Using Australian proxy..."
agent.agent.set_proxy(ENV["MORPH_AUSTRALIAN_PROXY"])
end

result = JSON.parse(page.body)

result.each do |a|
date_received = parse_date(a["rec_dte"])
if date_received >= Date.today - 30
if date_received && date_received >= Date.today - 30
record = {
"council_reference" => a["fmt_acc2"],
"address" => a["prm_adr"] + ", NSW",
Expand Down

0 comments on commit 881dbce

Please sign in to comment.