Scrapy Stackdriver (unofficial)

This Scrapy extension adds Google StackDriver logging to the Scrapy spiders. It simply attaches the Google handler to the spider's built-in logger, so you don't have to change your spider's code.

Logs will appear on the StackDriver log page for your project, under the name scraper.<spider name>.

Configuration

$ pip install [...]

Add to your settings.py

STACKDRIVER_ENABLED = True
STACKDRIVER_PROJECT_ID = '<your google cloud project>'

EXTENSIONS = {
    'scrapy_stackdriver.StackDriverLogger': 500
}

Settings

• STACKDRIVER_ENABLED controls whether the extension is active
• STACKDRIVER_PROJECT_ID should be your Google project id

As per other Google Cloud projects, you will also need to appropriately set the GOOGLE_APPLICATION_CREDENTIALS variable for your environment.