Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

logging level issues in 1.0.1 #1347

Closed
ileodo opened this issue Jul 9, 2015 · 8 comments
Closed

logging level issues in 1.0.1 #1347

ileodo opened this issue Jul 9, 2015 · 8 comments

Comments

@ileodo
Copy link

@ileodo ileodo commented Jul 9, 2015

Under the following setting up:

in setting.py:
LOG_ENABLED = False

in script to start the crawler:

configure_logging()
process = CrawlerProcess(get_project_settings())
process.crawl('retriever')
process.start()

in crawler:

logging.debug("PC get page [%s]:- %s" % (item['id'], item['url']))

There are still a lot of logging info.
even:

2015-07-09 23:29:20 [scrapy] DEBUG: Scraped from <200 http://www.example.com>

Is is a bug?

@kmike
Copy link
Member

@kmike kmike commented Jul 9, 2015

I think it is a documentation bug. Check http://doc.scrapy.org/en/latest/topics/practices.html#run-from-script - you need to run configure_logging() only for CrawlerRunner, not for CrawlerProcess. But configure_logging docstring says

configure_logging is automatically called when using Scrapy commands, but needs to be called explicitly when running custom scripts. In that case, its usage is not required but it’s recommended.

@SiddharthaAnand
Copy link

@SiddharthaAnand SiddharthaAnand commented Jan 25, 2019

@kmike Can I work on this issue?

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Mar 8, 2019

@SiddharthaAnand Have you changed your mind?

@anuja-kelkar
Copy link

@anuja-kelkar anuja-kelkar commented Mar 24, 2019

@kmike Can I work on this?

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Mar 25, 2019

@anuja-kelkar Please, go ahead.

@Vandenn
Copy link
Contributor

@Vandenn Vandenn commented May 2, 2019

Logging is currently behaving as expected when LOG_ENABLED is set to False using the following script in a Python 3.6 environment. Should this issue be closed? Kindly correct me if I'm making a wrong assumption about this.

import scrapy
from scrapy.crawler import CrawlerProcess


class MySpider(scrapy.Spider):
    name = 'spider'

process = CrawlerProcess({
    'LOG_ENABLED': False
})

process.crawl(MySpider)
process.start()

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented May 2, 2019

@Vandenn You are right, that’s why @kmike said it’s a documentation problem. The original poster, @ileodo, is calling configure_logging(), as the current documentation suggests, but that should not be used with CrawlerProcess, only with CrawlerRunner.

@Vandenn
Copy link
Contributor

@Vandenn Vandenn commented May 2, 2019

@Gallaecio Thanks for the clarification. I'll open a PR soon that updates the documentation of scrapy.utils.log.configure_logging to specify that the function should only be used with CrawlerRunner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

7 participants