Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+1] more stats for RobotsTxtMiddleware #3100

Merged
merged 1 commit into from Feb 8, 2018
Merged

[MRG+1] more stats for RobotsTxtMiddleware #3100

merged 1 commit into from Feb 8, 2018

Conversation

@kmike
Copy link
Member

@kmike kmike commented Jan 30, 2018

What do you think about adding mor stats for robots.txt middleware?

  • robotstxt/forbidden is a number of requests forbidden by robots.txt;
  • robotstxt/request_count is a number of requests to fetch robots.txt files;
  • robotstxt/response_count and robotstxt/response_status_count/.. are robots.txt response counts;
  • robotstxt/unicode_error_count is a number of robots.txt files with broken unicode;
  • robotstxt/exception_count/.. is a number of exceptions (download errors, parse errors).
@kmike kmike added the discuss label Jan 30, 2018
@codecov
Copy link

@codecov codecov bot commented Jan 30, 2018

Codecov Report

Merging #3100 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3100      +/-   ##
==========================================
+ Coverage   84.55%   84.56%   +0.01%     
==========================================
  Files         164      164              
  Lines        9291     9300       +9     
  Branches     1381     1383       +2     
==========================================
+ Hits         7856     7865       +9     
  Misses       1177     1177              
  Partials      258      258
Impacted Files Coverage Δ
scrapy/downloadermiddlewares/robotstxt.py 100% <100%> (ø) ⬆️
@lopuhin lopuhin changed the title more stats for RobotsTxtMiddleware [MRG+1] more stats for RobotsTxtMiddleware Feb 6, 2018
@lopuhin
Copy link
Member

@lopuhin lopuhin commented Feb 6, 2018

@kmike looks good 👍

@lopuhin lopuhin merged commit eff4692 into master Feb 8, 2018
2 checks passed
2 checks passed
codecov/patch 100% of diff hit (target 84.55%)
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@kmike kmike deleted the robots-stats branch Feb 14, 2018
@kmike kmike added this to the v1.6 milestone Jul 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants