No description or website provided.
Latest commit 2652a20 Dec 16, 2014 @dmclain Merge pull request #4 from summera/psycopg2-interface-error-fix
Reconnect to DB if connection closed
Failed to load latest commit information.
scrapy_heroku Catch psycopg2.OperationalError Dec 15, 2014
.gitignore build package properly Nov 16, 2012
LICENSE BSD, yo Nov 16, 2012 build package properly Nov 16, 2012 Update readme Nov 20, 2014 we need to connect to postgres Nov 16, 2012


A package to assist with running scrapy on heroku. This is accomplished by providing a custom application configuration at that launches the scrapyd web service using the PORT environment variable and a multi-process work queue implemented on a Postgres database specified by the DATABASE_URL environment variable.


Create a git repo that has a scrapy project at the root (scrapy.cfg should be at the top level). Edit your scrapy.cfg to include the following::

application =

url = http://<YOUR_HEROKU_APP_NAME>
username = <A_USER_NAME>
password = <A_PASSWORD>

Add a requirements.txt file that includes scrapy, scrapy-heroku, and scrapyd. It is strongly recommended that you version pin scrapy-heroku as well as the version of scrapy that your project is developed against (pip freeze > requirements.txt).

For Example:

# requirements.txt

Finally create a Procfile that consists of::

web: scrapyd

Make sure you have a postgres database with the DATABASE_URL env parameter set.