Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove unnecessary regexes #168

Merged
merged 3 commits into from
Jan 26, 2017
Merged

Remove unnecessary regexes #168

merged 3 commits into from
Jan 26, 2017

Conversation

MaxGiting
Copy link
Collaborator

@MaxGiting MaxGiting commented Jan 26, 2017

8 Regexes removed that were caught with the generic regex
7 Regexes remove that are now caught with the github\.com\/ regex.

This could lead to less coverage in some cases if user agents don't contain the generic catchall. For example I couldn't use the github regex to catch all CommaFeed useragents. But there is no way of knowing this without filling in more tests.

CommaFeed/2.4.0-SNAPSHOT (https://github.com/Athou/commafeed)
CommaFeed/2.3.0-SNAPSHOT (https://www.commafeed.com)

Every unique user agent we find should be put into tests to keep issues like this from happening. But we can't put every variation of bot user agent into the regex list as there would be thousands and slow this library down to a crawl. We have to strike a balance.

@MaxGiting MaxGiting merged commit d2dbf5e into master Jan 26, 2017
@MaxGiting MaxGiting deleted the dev branch January 29, 2017 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant