A Scrapy pipeline module to persist items to a postgres table automatically.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
pgpipeline actually only update Aug 14, 2017
.gitignore
LICENSE ii Aug 14, 2017
Makefile ii Aug 14, 2017
README.md ii Aug 14, 2017
requirements.txt
setup.py ii Aug 14, 2017

README.md

Pgpipeline: automatic postgres pipeline for Scrapy

A Scrapy pipeline module to persist items to a postgres table automatically.

Quick Start

Here's an example showing automatic item pipeline, with a custom JSONB field.

# settings.py
from sqlalchemy.dialects.postgresql import JSONB

ITEM_PIPELINES = {
    'pgpipeline.PgPipeline': 300,
}

PG_PIPELINE = {
    'connection': 'postgresql://localhost:5432/scrapy_db',
    'table_name': 'demo_items',
    'pkey': 'item_id',
    'ignore_identical': ['item_id', 'job_id'],
    'types': {
        'some_data': JSONB
    },
    'onconflict': 'upsert'
}

All columns, tables, and indices are automatically created.

  • pkey: a primary key for this item (other than database-generated id)
  • ignore_identical: these are a set of fields by which we identify duplicates and skip insert.
  • types: keys specified here will be using the type given, otherwise types are guessed.
  • onconflict: upsert|ignore|non-null - ignore will skip inserting on conflict and upsert will update. non-null will upsert only values that are not None and thus avoid removing existing values.

Developers

Set up a development environment

$ pip install -r requirements.txt

Development

  • Dependencies: list them in requirements.txt

Release

  • Dependencies: list them in setup.py under install_requires:
install_requires=['peppercorn'],

Then:

$ make dist && make release

Contributing

Fork, implement, add tests, pull request, get my everlasting thanks and a respectable place here :).

Thanks:

To all Contributors - you make this happen, thanks!

Copyright

Copyright (c) 2017 Dotan Nahum @jondot. See LICENSE for further details.