diff --git a/README.md b/README.md index 2ff20c3..7290a02 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,14 @@ # New Orleans real estate -[http://vault.thelensnola.org/realestate](http://vault.thelensnola.org/realestate) +[https://vault.thelensnola.org/realestate](https://vault.thelensnola.org/realestate) This app scrapes the latest property sales in New Orleans, stores the records in a database and publishes the results with a map. -[![Build Status](https://travis-ci.org/TheLens/realestate.svg?branch=master)](https://travis-ci.org/TheLens/realestate) [![Documentation Status](https://readthedocs.org/projects/realestate/badge/?version=latest)](https://readthedocs.org/projects/realestate/?badge=latest) [![Coverage Status](https://coveralls.io/repos/TheLens/realestate/badge.svg?branch=master)](https://coveralls.io/r/TheLens/realestate?branch=master) +[![Build Status](https://travis-ci.org/TheLens/realestate.svg?branch=master)](https://travis-ci.org/TheLens/realestate) [![Coverage Status](https://coveralls.io/repos/TheLens/realestate/badge.svg?branch=master)](https://coveralls.io/r/TheLens/realestate?branch=master) -Documentation: https://realestate.readthedocs.org/ - -Issues: https://github.com/TheLens/realestate/issues - -Tests: https://travis-ci.org/TheLens/realestate - -Testing coverage: https://coveralls.io/r/TheLens/realestate +- Issues: https://github.com/TheLens/realestate/issues +- Tests: https://travis-ci.org/TheLens/realestate +- Testing coverage: https://coveralls.io/r/TheLens/realestate #### Dependencies @@ -23,49 +19,37 @@ Testing coverage: https://coveralls.io/r/TheLens/realestate * SQLAlchemy * Virtualenvwrapper/virtualenv -#### Setup - -(Not yet sure if this setup will work or not.) - -`git clone https://github.com/TheLens/realestate.git` - -`cd` into the `realestate` directory. - -`pip install -r requirements.txt` - -`createdb realestate` - -`psql realestate < backups/realestate.sql` # For now, only available to The Lens employees. Access backup SQL file on S3. - #### Daily scripts -This will scrape, build, geocode, clean and publish the previous day's sales. These are summarized in `scripts/main.sh`, which is run on a cron job every night. - -`python realestate/lib/scrape.py` - -`python scripts/initialize.py` - -There is also a cron task to run `scripts/backups.sh`, which creates a datestamped database backup on the server and then copies the file to S3. - -#### Environment variables - -Create environment variables in `~/.virtualenvs/realestate/bin/postactivate` locally. Do the same at `/home/ubuntu/.virtualenvs/realestate/bin/postactivate` on the server, but first remove "export" in each line. +Every day, these two commands are run to scrape and then build, geocode, clean and publish the previous day's sales. These are summarized in `scripts/main.sh`, which is run on a cron job every night. ```bash -export SERVER_ENGINE='postgresql://myuser:mypass@localhost/realestate' - -export SERVER_CONNECTION='dbname=realestate user=myuser password=mypass' - -export LRD_USERNAME='MyLandRecordsDivisionUsername' +python realestate/lib/scrape.py +python scripts/initialize.py +``` -export LRD_PASSWORD='MyLandRecordsDivisionPassword' +Occasionally, due to bugs or password expiration, you will need to scrape and build beyond the previous day. You can specify the date range for both commands by using command-line arguments. -export DASHBOARD_USERNAME='myuser' +```bash +# +python realestate/lib/scrape.py 2016-05-01 2016-05-05 +python scripts/initialize.py 2016-05-01 2016-05-05 +``` -export DASHBOARD_PASSWORD='mypass' +There is also a cron task to run `scripts/backups.sh`, which creates a date-stamped database backup on the server and then copies the file to S3. This runs every night. -export REAL_ESTATE_GMAIL_USERNAME=tthoren@thelensnola.org +#### Environment variables -export REAL_ESTATE_GMAIL_PASSWORD=mypass +Create environment variables in `~/.virtualenvs/realestate/bin/postactivate` locally. Do the same at `/home/ubuntu/.virtualenvs/realestate/bin/postactivate` on the server. You will need to include two lines in the server's postactivate file so that Upstart can find and use the variables. +```bash +REAL_ESTATE_LRD_USERNAME='MyLandRecordsDivisionUsername' +REAL_ESTATE_LRD_PASSWORD='MyLandRecordsDivisionPassword' +REAL_ESTATE_DATABASE_USERNAME='MyDatabaseUsername' +REAL_ESTATE_DATABASE_PASSWORD='MyDatabasePassword' + +export REAL_ESTATE_LRD_USERNAME='MyLandRecordsDivisionUsername' +export REAL_ESTATE_LRD_PASSWORD='MyLandRecordsDivisionPassword' +export REAL_ESTATE_DATABASE_USERNAME='MyDatabaseUsername' +export REAL_ESTATE_DATABASE_PASSWORD='MyDatabasePassword' ```