Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Pgpipeline: automatic postgres pipeline for Scrapy

A Scrapy pipeline module to persist items to a postgres table automatically.

Quick Start

Here's an example showing automatic item pipeline, with a custom JSONB field.

# settings.py
from sqlalchemy.dialects.postgresql import JSONB

ITEM_PIPELINES = {
    'pgpipeline.PgPipeline': 300,
}

PG_PIPELINE = {
    'connection': 'postgresql://localhost:5432/scrapy_db',
    'table_name': 'demo_items',
    'pkey': 'item_id',
    'ignore_identical': ['item_id', 'job_id'],
    'types': {
        'some_data': JSONB
    },
    'onconflict': 'upsert'
}

All columns, tables, and indices are automatically created.

  • pkey: a primary key for this item (other than database-generated id)
  • ignore_identical: these are a set of fields by which we identify duplicates and skip insert.
  • types: keys specified here will be using the type given, otherwise types are guessed.
  • onconflict: upsert|ignore|non-null - ignore will skip inserting on conflict and upsert will update. non-null will upsert only values that are not None and thus avoid removing existing values.

Developers

Set up a development environment

$ pip install -r requirements.txt

Development

  • Dependencies: list them in requirements.txt

Release

  • Dependencies: list them in setup.py under install_requires:
install_requires=['peppercorn'],

Then:

$ make dist && make release

Contributing

Fork, implement, add tests, pull request, get my everlasting thanks and a respectable place here :).

Thanks:

To all Contributors - you make this happen, thanks!

Copyright

Copyright (c) 2017 Dotan Nahum @jondot. See LICENSE for further details.

About

A Scrapy pipeline module to persist items to a postgres table automatically.

Resources

License

Releases

No releases published

Packages

No packages published
You can’t perform that action at this time.