Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG] Fix for #612 + small tests cleanup #656

Merged
merged 4 commits into from Mar 19, 2014
Merged

Conversation

@kmike
Copy link
Member

@kmike kmike commented Mar 18, 2014

Log looks like this now:

2014-03-19 03:56:30+0600 [scrapy] Crawled (402) <GET http://localhost:8998/status?n=402> (referer: None)
2014-03-19 03:56:30+0600 [scrapy] Ignoring response <402 http://localhost:8998/status?n=402>: HTTP status code is not handled or not allowed

I haven't put a test spider to scrapy.tests.spiders because it is unlikely be useful for other tests.

docrawl function was copy-pasted several times so I added it to scrapy.utils.test. In test_crawl crawler is now configured immediately instead of being configured via yield defer.maybeDeferred(self.configure) in crawler.start(), but it doesn't seem to affect anything.

@kmike kmike changed the title Fix for #612 + small tests cleanup [MRG] Fix for #612 + small tests cleanup Mar 18, 2014
dangra added a commit that referenced this pull request Mar 19, 2014
[MRG] Fix for #612 + small tests cleanup
@dangra dangra merged commit 3df6946 into scrapy:master Mar 19, 2014
1 check passed
1 check passed
default The Travis CI build passed
Details
@nramirezuy
Copy link
Contributor

@nramirezuy nramirezuy commented Mar 19, 2014

Wasn't this log line removed some time ago because a issue? Maybe it should be configurable, IgnoreRequest can accept an optional argument to be logged or not; or there can be a list of exceptions to not log.

@dangra
Copy link
Member

@dangra dangra commented Mar 19, 2014

@nramirezuy are you sure it was this log line ? I recall about offsitemiddleware logging but not for httperror middleware.

@nramirezuy
Copy link
Contributor

@nramirezuy nramirezuy commented Mar 19, 2014

@dangra I just remember a IgnoreRequest, I don't remember from where module it comes. Does it matters? We already know that those requests are banned by this middleware, maybe new comers want to see this but at some point it just spam.

@kmike
Copy link
Member Author

@kmike kmike commented Mar 19, 2014

This is about dropping the responses, and AFAIK there was no indication they are dropped: log just said Crawled (402) ..., but spider callback was not called and the response was not processed.

@kmike kmike deleted the kmike:httperror-logging branch Jun 24, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.