Skip to content
Scraper (web crawler) for vehicle classified websites in Qatar written in Ruby on Rails
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
app
config
db
doc
lib
public
script
test
vendor
.gitignore
.travis.yml
Gemfile
Gemfile.lock
README.rdoc
Rakefile
config.ru

README.rdoc

Qscrape is a Ruby on Rails app that scrapes vehicle classified ads from websited in Qatar. It comes with a web interface to visualize ads, search in them, favorite and comment on any. There are also 2 rake tasks to scrape and notify, respectively.

Installation

bundle

Scraping

rake scraper:scrape_ql  # QatarLiving

or to scrape from a specific page (0 based index):

rake scraper:scrape_ql[99] # starts from page 100

To scrape from QatarSale:

rake scraper:scrape_qs  # QatarSale

Notifying

To send an email notification if an ad matches your text criteria, run:

rake scraper:notify[criteria]

Note that you need to insert records for each recipient in the recipients table, for example:

rails console
> Recipient.create name: 'Your name', email: 'your@email'

Cron

You may put any of these 2 rake tasks in a cron job that runs every 10 or 15 minutes to periodically scrape and notify you. An example crontab file can found in file:lib/tasks/crontab To install:

crontab lib/tasks/crontab
You can’t perform that action at this time.