scrapy-http-pipeline

Just a simple Scrapy HTTP pipeline to POST your items to your server.

Usage

Install

pip install scrapy-http-pipeline

Configure scrapy settings.py

ITEM_PIPELINES = {
    'scrapyhttppipeline.scrapyhttppipeline.HttpPostPipeline': 500
}

# Url to your server, which accepts POST requests
HTTP_POST_PIPELINE_URL = 'http://localhost:8080/items'

# Any custom headers you want to add, e.g. authentication
HTTP_POST_PIPELINE_HEADERS = {
    'X-Authorization': 'xxx'
}

# If you want to send more items at once (and have less HTTP POST requests incoming.)
# If True items will be send as [{key1:val1},{key1:val1}] instead of {key1:val1}
HTTP_POST_PIPELINE_BUFFERED = False
HTTP_POST_PIPELINE_BUFFER_SIZE = 100

Developing

Package requirements are handled using pip. To install them do

pip install -r requirements.txt

Tests

Testing is set up using pytest and coverage is handled with the pytest-cov plugin.

Run your tests with py.test in the root directory.

Coverage is ran by default and is set in the pytest.ini file. To see an html output of coverage open htmlcov/index.html after running the tests.

Travis CI

There is a .travis.yml file that is set up to run your tests for python 2.7 and python 3.3-3.7, should you choose to use it.