===================
Course Crawler
==================

Columbia's data formats are subject to change, and we cannot guarantee that this
program will be compatible with future formats.

Binary Dependencies
------------

- Ruby 1.8.7>=
- MySQL 5.0>=


Installation and Setup
------------

Before beginning, make sure you have Ruby 1.8.7 or later and MySQL 5.0 or
later installed.

- Copy `config.yaml.default` to `config.yaml`
- Complete the fields in `config.yaml`
- run `gem install bundler`
- run `bundle install`


Crawling
------------

Once you've set up the application, run the app.rb file in the root directory
to begin the crawling process.

HTTP requests are made in parallel using the Typhoeus gem.


Exporting data
------------

Data is stored in a local database, which can be exported to a text file
in SQL format using the command:

`rake db:export`

The result is stored in the local directory in a file named "data.sql"

Searching
----------
After updating the database, you need to run sphinx to update your indices so
that searching works: rake thinking_sphinx:index RAILS_ENV=production