Scrapy pipeline to communicate with amazon sqs.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
sqspipeline
.gitignore
LICENSE.txt
README.md
requirementst.txt
setup.cfg
setup.py

README.md

Summary

Simple pipeline to write to Amazon's Simple Queue Service (SQS).

The pipeline will write json serialized items to the queue specified in your scrapy settings file.

Configuration

To configure the pipeline add the following lines to settings.py

AWS_ACCESS_KEY = '<your_access_key>'
AWS_SECRET_KEY = '<your_secret_key>'
AWS_REGION = '<your_aws_region>'
SQS_QUEUE_NAMES = ['<your_sqs_queue_name>']

ITEM_PIPELINES = {
    'sqspipeline.BotoPipeline': 300
}

# Optional
# Default = False
# Raise CloseSpider exception if there are still items in the queue
RAISE_IF_QUEUE_NOT_EMPTY = True

Installation

pip install scrapy-sqs-pipeline