Skip to content

Latest commit

 

History

History
129 lines (88 loc) · 3.75 KB

logging.rst

File metadata and controls

129 lines (88 loc) · 3.75 KB
Raw

Logging

Scrapy provides a logging facility which can be used through the scrapy.log module. The current underlying implementation uses Twisted logging but this may change in the future.

The logging service must be explicitly started through the scrapy.log.start function to catch the top level Scrapy's log messages. On top of that, each crawler has its own independent log observer (automatically attached when it's created) that intercepts its spider's log messages.

Log levels

Scrapy provides 5 logging levels:

  1. ~scrapy.log.CRITICAL - for critical errors
  2. ~scrapy.log.ERROR - for regular errors
  3. ~scrapy.log.WARNING - for warning messages
  4. ~scrapy.log.INFO - for informational messages
  5. ~scrapy.log.DEBUG - for debugging messages

How to set the log level

You can set the log level using the --loglevel/-L command line option, or using the LOG_LEVEL setting.

How to log messages

Here's a quick example of how to log a message using the WARNING level:

from scrapy import log
log.msg("This is a warning", level=log.WARNING)

Logging from Spiders

The recommended way to log from spiders is by using the Spider ~scrapy.spider.Spider.log method, which already populates the spider argument of the scrapy.log.msg function. The other arguments are passed directly to the ~scrapy.log.msg function.

scrapy.log module

scrapy.log

start(logfile=None, loglevel=None, logstdout=None)

Start the top level Scrapy logger. This must be called before actually logging any top level messages (those logged using this module's ~scrapy.log.msg function instead of the Spider.log <scrapy.spider.Spider.log> method). Otherwise, messages logged before this call will get lost.

param logfile

the file path to use for logging output. If omitted, the LOG_FILE setting will be used. If both are None, the log will be sent to standard error.

type logfile

str

param loglevel

the minimum logging level to log. Available values are: CRITICAL, ERROR, WARNING, INFO and DEBUG.

param logstdout

if True, all standard output (and error) of your application will be logged instead. For example if you "print 'hello'" it will appear in the Scrapy log. If omitted, the LOG_STDOUT setting will be used.

type logstdout

boolean

msg(message, level=INFO, spider=None)

Log a message

param message

the message to log

type message

str

param level

the log level for this message. See topics-logging-levels.

param spider

the spider to use for logging this message. This parameter should always be used when logging things related to a particular spider.

type spider

~scrapy.spider.Spider object

CRITICAL

Log level for critical errors

ERROR

Log level for errors

WARNING

Log level for warnings

INFO

Log level for informational messages (recommended level for production deployments)

DEBUG

Log level for debugging messages (recommended level for development)

Logging settings

These settings can be used to configure the logging:

  • LOG_ENABLED
  • LOG_ENCODING
  • LOG_FILE
  • LOG_LEVEL
  • LOG_STDOUT