Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add flag to disable Scraped debug messages #3984

Closed
krisztian-toth opened this issue Aug 28, 2019 · 2 comments · Fixed by #3987
Closed

Add flag to disable Scraped debug messages #3984

krisztian-toth opened this issue Aug 28, 2019 · 2 comments · Fixed by #3987

Comments

@krisztian-toth
Copy link
Contributor

krisztian-toth commented Aug 28, 2019

Summary

Add a flag to settings (e.g. LOG_SCRAPED_ENABLED) to be able to disable the logging of each scraped Item on DEBUG level.

Motivation

Crawling with Scrapy on log level DEBUG provides a lot of useful context during development. The only problem is the "scraped" message (constructed in scrapy.logformatter.LogFormatter.scraped) - it obfuscates the log entirely in the command line and can be really frustrating in general. Adding a flag to disable this log message (with a default value so that the current behaviour doesn't change) would be really useful.

Describe alternatives you've considered

Right now you either put up with it or use INFO level instead which makes you lose a lot of useful log entries available in DEBUG level.

@elacuesta
Copy link
Member

elacuesta commented Aug 28, 2019

Seems like a valid request, I have seen my logs flooded with those messages as well.
I believe it shouldn't be too hard to implement, probably just an if check around these lines.

@krisztian-toth
Copy link
Contributor Author

krisztian-toth commented Aug 28, 2019

Thanks @elacuesta, that's what I thought as well, I'd be happy to pick this up when I have a bit of time.

krisztian-toth pushed a commit to krisztian-toth/scrapy that referenced this issue Aug 30, 2019
krisztian-toth pushed a commit to krisztian-toth/scrapy that referenced this issue Aug 30, 2019
krisztian-toth pushed a commit to krisztian-toth/scrapy that referenced this issue Sep 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants