I thought that it could be redundant - you'll get the same exception names as in downloader, e.g. 'downloader/exception_type_count/twisted.internet.error.ConnectionRefusedError': 155 and 'retry/twisted.internet.error.ConnectionRefusedError': 155; downloader/response_status_count/ will be also duplicated - it could be 20x lines in stats with very little new information.
Yeah, this is true. My point is that you can spot it without adding this information to stats: status codes and exceptions to retry are static, they don't depend on website - you can check settings and EXCEPTIONS_TO_RETRY list. You check retry/count, notice that the value is high (=> there is some issue with crawling this website), and then investigate it further: check response status codes and downloader exceptions. So this PR does this minimal change which allows to spot and debug the problem.
I'm not opposed to adding more information to stats, it is just a bit more controversial IMHO (#2173 becomes more verbose). But more stats could make it easier to debug retry issues, that's true.
I don't mind having verbose stats if it's to debug problems. Having to look at the code for EXCEPTIONS_TO_RETRY and default RETRY_HTTP_CODES (and settings to double check if it's not the default) feels not-straightforward.
I'm not sure I would spot retry/count more than high downloader/exception_type_count values.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.