Google Stackdriver Logging extension for Scrapy
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
scrapy_stackdriver
tests
.gitignore
LICENSE
README.md
requirements.txt
setup.py

README.md

Scrapy Stackdriver (unofficial)

This Scrapy extension adds Google StackDriver logging to the Scrapy spiders. It simply attaches the Google handler to the spider's built-in logger, so you don't have to change your spider's code.

Logs will appear on the StackDriver log page for your project, under the name scraper.<spider name>.

Configuration

$ pip install [...]

Add to your settings.py

STACKDRIVER_ENABLED = True
STACKDRIVER_PROJECT_ID = '<your google cloud project>'

EXTENSIONS = {
    'scrapy_stackdriver.StackDriverLogger': 500
}

Settings

  • STACKDRIVER_ENABLED controls whether the extension is active
  • STACKDRIVER_PROJECT_ID should be your Google project id

As per other Google Cloud projects, you will also need to appropriately set the GOOGLE_APPLICATION_CREDENTIALS variable for your environment.