AWS DynamoDB pipeline for Scrapy
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.gitignore
.travis.yml
LICENSE
MANIFEST.in
README.rst
requirements.txt
scrapy_dynamodb.py
setup.py

README.rst

scrapy-dynamodb

scrapy-dynamodb allows you to export your items to a DynamoDB database.

Installation

You can install scrapy-dynamodb either via the Python Package Index (PyPI) or from source.

To install using pip,:

$ pip install scrapy-dynamodb

To install using easy_install,:

$ easy_install scrapy-dynamodb

Downloading and installing from source

Download the latest version of scrapy-dynamodb from http://pypi.python.org/pypi/scrapy-dynamodb/

You can install it by doing the following,:

$ tar xvfz scrapy-dynamodb-0.0.0.tar.gz
$ cd scrapy-dynamodb-0.0.0
# python setup.py install # as root

Using the development version

You can clone the git repository by doing the following:

$ git clone git://github.com/acordiner/scrapy-dynamodb.git

Using scrapy-dynamodb

To use scrapy-dynamodb in your Scrapy project, add the following line to your settings.py:

ITEM_PIPELINES = {
    'scraper.pipelines.DynamoDbPipeline': 1,
}

AWS_ACCESS_KEY_ID = '<aws access key id>'
AWS_SECRET_ACCESS_KEY = '<aws secret access key>'
DYNAMODB_PIPELINE_REGION_NAME = 'us-east-1'
DYNAMODB_PIPELINE_TABLE_NAME = 'my_table'

That's it!