[Robots.txt] Close groups of rules as defined in RFC 9309 #430
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ster rules) - add unit test to verify solution of crawler-commons#114
…gents, closes crawler-commons#390 [Robots.txt] Handle robots.txt with missing sections (and implicit master rules), fixes crawler-commons#114 - do not close rule blocks / groups on other directives than specified in RFC 9309: groups are only closed on a user-agent line at least one allow/disallow line was read before - set Crawl-delay independently from grouping, but never override or set the value for a specific agent using a value defined for the wildcard agent
…gents - fix unit test broken by introducing compliance with RFC 9309
sebastian-nagel
force-pushed
the
cc-390-114-robots-closing-rule-group
branch
from
35053f5
to
17e8544
Compare
…gents - must keep state whether Crawl-delay is already set for a specific agent as separate variable - add unit test to ensure that no already set Crawl-delay is overridden by a (lower) value of another agent
jnioche
approved these changes
Jul 11, 2023
|
Thanks for the review, @jnioche! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses:
Changes:
This PR should unblock #245 (all unit tests added there should now pass).