Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Summary

Simple pipeline to write to Amazon's Simple Queue Service (SQS).

The pipeline will write json serialized items to the queue specified in your scrapy settings file.

Configuration

To configure the pipeline add the following lines to settings.py

AWS_ACCESS_KEY = '<your_access_key>'
AWS_SECRET_KEY = '<your_secret_key>'
AWS_REGION = '<your_aws_region>'
SQS_QUEUE_NAMES = ['<your_sqs_queue_name>']

ITEM_PIPELINES = {
    'sqspipeline.BotoPipeline': 300
}

# Optional
# Default = False
# Raise CloseSpider exception if there are still items in the queue
RAISE_IF_QUEUE_NOT_EMPTY = True

Installation

pip install scrapy-sqs-pipeline

About

Scrapy pipeline to communicate with amazon sqs.

Resources

License

Releases

No releases published

Packages

No packages published

Languages

You can’t perform that action at this time.