Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+1] Adding requests referer to RFPDupeFilter log messages #3588

Merged
merged 2 commits into from Jan 28, 2019

Conversation

@ejulio
Copy link
Contributor

@ejulio ejulio commented Jan 18, 2019

I just changed the log message when DUPEFILTER_DEBUG is active because it helps knowing which request is generating the duplicates (I'd this problem recently).

Also, I added some tests for the log method to check the proper messages and stats.

Copy link
Contributor

@victor-torres victor-torres left a comment

LGTM

@dangra dangra changed the title Adding requests referer to RFPDupeFilter log messages [MRG+1] Adding requests referer to RFPDupeFilter log messages Jan 25, 2019
@dangra dangra added this to the v1.6 milestone Jan 25, 2019
@dangra
Copy link
Member

@dangra dangra commented Jan 25, 2019

@kmike if you don't see any objections, I'll merge and ship it in 1.6.0

@dangra dangra merged commit c4f5760 into scrapy:master Jan 28, 2019
2 checks passed
2 checks passed
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@dangra
Copy link
Member

@dangra dangra commented Jan 28, 2019

thanks Julio.

msg = "Filtered duplicate request: %(request)s"
self.logger.debug(msg, {'request': request}, extra={'spider': spider})
msg = "Filtered duplicate request: %(request)s (referer: %(referer)s)"
args = {'request': request, 'referer': referer_str(request) }

This comment has been minimized.

@kmike

kmike Jan 28, 2019
Member

Hey! Sorry for delay. No objections to me, a nice feature. I was just wondering if Referer should be added to the next message as well, when debug is False.

This comment has been minimized.

@ejulio

ejulio Jan 28, 2019
Author Contributor

Fair point.
I just added to keep the same behavior, but it makes sense to log referer only in debug mode.

@ejulio ejulio deleted the ejulio:dupefilter-debug branch Jan 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants