Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+1] more stats for RobotsTxtMiddleware #3100

merged 1 commit into from Feb 8, 2018

[MRG+1] more stats for RobotsTxtMiddleware #3100

merged 1 commit into from Feb 8, 2018


Copy link

@kmike kmike commented Jan 30, 2018

What do you think about adding mor stats for robots.txt middleware?

  • robotstxt/forbidden is a number of requests forbidden by robots.txt;
  • robotstxt/request_count is a number of requests to fetch robots.txt files;
  • robotstxt/response_count and robotstxt/response_status_count/.. are robots.txt response counts;
  • robotstxt/unicode_error_count is a number of robots.txt files with broken unicode;
  • robotstxt/exception_count/.. is a number of exceptions (download errors, parse errors).

@kmike kmike added the discuss label Jan 30, 2018
Copy link

codecov bot commented Jan 30, 2018

Codecov Report

Merging #3100 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3100      +/-   ##
+ Coverage   84.55%   84.56%   +0.01%     
  Files         164      164              
  Lines        9291     9300       +9     
  Branches     1381     1383       +2     
+ Hits         7856     7865       +9     
  Misses       1177     1177              
  Partials      258      258
Impacted Files Coverage Δ
scrapy/downloadermiddlewares/ 100% <100%> (ø) ⬆️

@lopuhin lopuhin changed the title more stats for RobotsTxtMiddleware [MRG+1] more stats for RobotsTxtMiddleware Feb 6, 2018
Copy link

lopuhin commented Feb 6, 2018

@kmike looks good 👍

@lopuhin lopuhin merged commit eff4692 into master Feb 8, 2018
@kmike kmike deleted the robots-stats branch February 14, 2018 15:51
@kmike kmike added this to the v1.6 milestone Jul 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

Successfully merging this pull request may close these issues.

None yet

2 participants