No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore
.travis.yml
LICENSE.txt
MANIFEST
Makefile
README.md
requirements.txt
scrapy_firebase.py
setup.py

README.md

scrapy-firebase

Firebase pipeline for Scrapy.

Installation

Install via pip:

pip install scrapy-firebase

Configuration

Basic configuration example

Follow the steps on this guide, once you downloaded the json with firebase secrets, convert it to base64. On macos you can use the follwing command line to convert and copy to your clipboard:

cat firebase_secrets.json | openssl base64 | pbcopy

Add scrapy-firebase to your projects settings.py file and setup some variables.

ITEM_PIPELINES = [
  'scrapy_firebase.FirebasePipeline',
]

FIREBASE_SECRETS = """
  YOUR BASE64 ENCODED JSON HERE
"""

# Replace project-id to yours.
FIREBASE_DATABASE = 'https://project-id.firebaseio.com/'

# Insert an appropriate value.
FIREBASE_REF = ''

# To compose more robust child paths, you can add a list of properties.
FIREBASE_KEYS = ['uid', 'spider_name']