In our case, sometimes the item can be very large, in which case it being printed completely to the log is problematic -- specifically, on ScrapyingHub the log entry is refused and the exception is not visible.
Describe alternatives you've considered
Modify that file adding a generic error logformatter function.
The text was updated successfully, but these errors were encountered:
Summary
Allow users to specify the logging behaviour in case of errors in the item pipeline
Motivation
Looking at https://github.com/scrapy/scrapy/blob/3adf09bc4f1468f54a703481d6d7ebfe25b6f232/scrapy/core/scraper.py, it is possible to override the logging behaviour for the
DropItem
exception, but there is no equivalentself.logformatter.error
.In our case, sometimes the item can be very large, in which case it being printed completely to the log is problematic -- specifically, on ScrapyingHub the log entry is refused and the exception is not visible.
Describe alternatives you've considered
Modify that file adding a generic error logformatter function.
The text was updated successfully, but these errors were encountered: