Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scrapy LOG_LEVEL setting in Spider.custom_settings does not work #1612

Closed
YAmikep opened this issue Nov 23, 2015 · 13 comments
Closed

scrapy LOG_LEVEL setting in Spider.custom_settings does not work #1612

YAmikep opened this issue Nov 23, 2015 · 13 comments
Labels

Comments

@YAmikep
Copy link

@YAmikep YAmikep commented Nov 23, 2015

I set the LOG_LEVEL setting to INFO in the Spider class via the custom_settings attribute but I still see the DEBUG messages in the console.

When I set it on the settings.py file or via the command line option --loglevel, it works.

I thought any settings could be set via the custom_settings attribute. Is that a bug? (Scrapy 1.0.3 and python 2.7.10)

class TestSpider(scrapy.Spider):
    name = "Test"
    ...
    custom_settings = {
        'LOG_LEVEL': 'INFO',
    }
@jdemaeyer
Copy link
Contributor

@jdemaeyer jdemaeyer commented Nov 23, 2015

Looks like a bug, although support for setting LOG_LEVEL on the spider might be dropped in the future (#1580)

@curita Probably because we're using settings instead of self.settings in this line?

@nramirezuy
Copy link
Contributor

@nramirezuy nramirezuy commented Nov 23, 2015

I think changing LOG_LEVEL from the spider doesn't make too much sense anyways.

@YAmikep
Copy link
Author

@YAmikep YAmikep commented Nov 24, 2015

@nramirezuy Could you elaborate on your answer? I'm new to scrapy. Why it does not make sense to set the logging level on a Spider?
Where do you set it? On a project? What if I have different Spider?

Thanks

@nramirezuy
Copy link
Contributor

@nramirezuy nramirezuy commented Nov 24, 2015

@YAmikep It doesn't make sense because there are 2 clear stages, development and production. During development you will probably want to use DEBUG, but for production you will want INFO or WARNING. This 2 should be 2 completely different environments, making the setting to have no sense.

What if you need to enable DEBUG on production for some reason; well then you can use command line to set the LOG_LEVEL.

@YAmikep
Copy link
Author

@YAmikep YAmikep commented Nov 25, 2015

Thanks @nramirezuy . I see what you meant now.
I agree and actually, only the command line option makes sense. Changing it at the project setting does not make sense either.

@nramirezuy
Copy link
Contributor

@nramirezuy nramirezuy commented Nov 25, 2015

@YAmikep I guess you are right. But you can cheat it using local_settings.py.

@jackie2013
Copy link

@jackie2013 jackie2013 commented Jan 3, 2016

@nramirezuy I set LOG_FILE setting in custom_settings for every Spider to get a different log, but it not work, what should i do? Thanks.

@redapple redapple added the logging label Jan 27, 2016
@djunzu
Copy link
Contributor

@djunzu djunzu commented Sep 16, 2016

@nramirezuy , I agree with you: development (DEBUG) is different from production (INFO). But I am facing a particular case...

I run multiple spiders using CrawlerProcess, each spider for one website. One of them is very good example of poorly coded html we find in the wild! It is a really f* mess. Plus, it is always changing. So, almost every time a run my crawler (once every 45 days) I find out this particular spider is not running correctly because of the recent changes. Then I need to run it alone in development to be able to fix it.

If I could set just this spider to debug mode, my production would be in info mode and I would know how to fix this particular spider without doing a complete run in debug mode.

My other option is to always run this spider alone. But my workflow would be much more consistent if this bug was fixed.

@kmike kmike removed the wontfix label Sep 19, 2016
@kmike
Copy link
Member

@kmike kmike commented Sep 19, 2016

+1 to fix it; it looks like a bug for me.

@shyandsy
Copy link

@shyandsy shyandsy commented Oct 9, 2016

so how to solve this now?

I am running on python 3.5, and all the log information, such as info and debug goes into the log file.
but i only wanna the ERROR!

image
image

BUT
image

@djunzu
Copy link
Contributor

@djunzu djunzu commented Oct 11, 2016

@shyandsy , your question is not related to this issue even though it is about logging.

If you wanna error level in everywhere, just use the right setting: LOG_LEVEL

@shyandsy
Copy link

@shyandsy shyandsy commented Oct 12, 2016

@djunzu
hi i have the log level setting in settings.py
image

but the info, debug also wrote in log file

@ianhhhhhhhhe
Copy link

@ianhhhhhhhhe ianhhhhhhhhe commented Feb 19, 2017

About settings, i think maybe you setup but you don't know how to import or activate them. And that's why your logging doesn't work.

So here are some solutions might help.

You need to sure that settings url is corret in root/scrapy.cfg. Then you can set your logging in settings.py. I suggest at least setup LOG_FILE and LOG_LEVEL.

Then you can run scrapy crawl crawler_name

If you want to run the spider from a script, you will need from scrapy.utils.project import get_project_settings and from scrapy.utils.log import configure_logging . Then put configure_logging(get_project_settings) to where you need to activating logging.

lopuhin added a commit to lopuhin/scrapy that referenced this issue Feb 21, 2017
lopuhin added a commit to lopuhin/scrapy that referenced this issue Feb 21, 2017
lopuhin added a commit to lopuhin/scrapy that referenced this issue Mar 2, 2017
Now configure_logging is called each time a crawler is created,
this allows to respect all custom settings related to logging,
such as LOG_LEVEL, LOG_FILE, etc.
lopuhin added a commit to lopuhin/scrapy that referenced this issue Mar 2, 2017
Now configure_logging is called each time a crawler is created,
this allows to respect all custom settings related to logging,
such as LOG_LEVEL, LOG_FILE, etc.
@redapple redapple closed this in ef04cfd Mar 9, 2017
redapple added a commit that referenced this issue Mar 9, 2017
[MRG+1] Respect custom log level: fixes GH-1612
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants