Reduce false positives (those not caused by WAFs or bot detection) #2068
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The following targets were fixed: Archive[.]org CGTrader CNET Contently IFTTT Linktree xHamster The following targets were removed: HexRPG (auth wall) ModelHub (defunct) Oracle Communities (auth wall) ModelHub was not added to ./removed_sites.md as the platform itself is shutting down (and will therefore never return to Sherlock). The other removed targets were documented normally. BitcoinForum is currently down and suspected to be defunct. Since this is uncertain, however, a test condition was added to suppress false positives while allowing for normal operation upon the forum's return.
Closed
Error codes module expanded to support arrays of error codes rather than only one. Using this new functionality, Slides was set to error codes 404 (as standard) AND 204 (non standard), to accomodate for that website's odd edge case.
ppfeister
force-pushed
the
master
branch
2 times, most recently
from
040100e
to
b6564a8
Compare
1 task
Attempts were met with a Varnish error page presenting 54113 (possibly Fastly related). Change to User Agent necessary to avoid Varnish/Fastly issues. Change to Accept necessary to avoid infinite 302 redirection. Without BOTH of these changes, attempts will fail. Both changes being made also permit the use of status_code rather than message.
|
A lil bit larger of a pr than first expected........ I think that's all of em, though... Well, I hope that's all of em For general testing purposes, ppfeister:rc/combopoc currently reflects master with this, #2069, #2070, and #2092 all merged. (ppfeister:rc/combopoc itself has a messy merge history though -- don't use that branch for any merging upstream) Cheers. |
ppfeister
changed the title
I've noticed that many bot-detection pages are able to be avoided by using this UA. Unless there's a reason to stay on the old old old one, we may as well update it and reduce our WAF hits.
ppfeister
changed the title
|
Thank you so much @ppfeister for fixing all of this, I really appriciate it! |
This was referenced May 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The following targets were fixed or partially fixed:
Archive[.]org (eh. sometimes. it has a loading problem that causes other F+s... some were fixed)
Archive of Our Own
CGTrader
CNET
Contently
Eintracht Frankfurt Forum
GeeksForGeeks
Genius[.]com Artists
Genius[.]com Users
Gumroad
HackerNews
HackerRank
IFTTT
Kongregate
Linktree
OpenStreetMap
Pinkbike
Polymart (often hits bot detection, but can be bypassed with some proxies)
Slides
Splits[.]io
Strava
Telegram
xHamster
YandexMusic
eintracht
jeuxvideo
The following targets were removed:
BitcoinForum (likely defunct)
G2G
HexRPG (auth wall)
Metacritic
ModelHub (defunct)
Oracle Communities (auth wall)
Misc changes:
Default User Agent updated
ModelHub was not added to ./removed_sites.[md|json] as the platform itself is confirmed to be shutting down (and will therefore never return to Sherlock). The other removed targets were documented normally.
Fiverr, Euw, NationStates Nations, NationStates Regions, and a couple others remain occasionally problematic as they are behind WAFs and bot detection services. These WAF-induced false positives are resolved with sister PR #2069 and have partial support for a decent proxy being discussed in #2081.
Issues:
Fixes #904
Fixes #1966
Fixes #1999 (with sister PR #2069)
Fixes #2027 (with sister PR #2069)
Fixes #2071
Collateral (trumped or negated):
Closes #1843 // Removes jeuxvideo instead of applying a fix (a fix is being applied here).
Closes #2083 // Removes ModelHub, which is an action taken here. Would be negated by merge.
Closes #2096 // Fixes Archive[.]org, which is an action taken here. Would be negated by merge.