Skip to content
No description or website provided.
Ruby
Find file
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
db
lib
log
spec
storage
.gitignore
Gemfile
Gemfile.lock
README
Rakefile
required.rb
schema.sql
worker_bendmaps.rb
worker_bot.rb
worker_convert.rb

README

DeschutesCounty
Copyright Jared Folkins 2011

Description:

  This is a web scraping program I wrote to mine the
  deschutes county website public foreclosure records.
  I use this data to determine how many foreclosures
  are coming to market and understand better buying
  opportunities.

Options:

  --page=PAGE                  Crawl a specific page's tree. PAGE takes precedence over SKIP. Also, error will occur if YEAR is set.
  --skip=SKIP                  Skip forward a certain number of pages on the search results page.
  --year=YEAR                  Crawl through documents based by year.
  --limit=LIMIT                Limit the total number of search result pages to crawl. (primary used to refresh the dataset without having to crawl the whole site)
  --trace=TRACE                Enable Memprof to profile code.

Examples:

  $ruby worker_bot.rb -h

Desciptions:

--page: will go to the page with instrument_id 920850

  $ ruby worker_bot.rb --page 920850

--skip: will skip the first 15 pages of the search result

  $ ruby worker_bot.rb --skip 15

--year: only crawl the records for 2006

  $ ruby worker_bot.rb --year 2006

--limit: only crawl N number of search result pages 

  $ ruby worker_bot.rb --year 2006

--trace: enable memprof

  $ ruby worker_bot.rb --trace true

Mix and Match: search only the year 2001 and skip 5 pages

  $ ruby worker_bot.rb --year 2001 --skip 5
Something went wrong with that request. Please try again.