Permalink
Switch branches/tags
Nothing to show
Commits on Sep 30, 2017
  1. Add a full list of validated bots to skip the fake browser tests.

    Flameeyes committed Sep 30, 2017
    This should make it easier to maintain in the long run, and whitelists all
    of the yandex bots too.
  2. Implement better validation of all Yandex bots.

    Flameeyes committed Sep 30, 2017
    This should make it easier to filter out fake Yandex bots that may be
    avoiding the primary crawler.
  3. Add playamo to the list of banned referrer.

    Flameeyes committed Sep 30, 2017
    This appears to be all coming from the same IP too.
Commits on Aug 29, 2017
  1. Remove validation for gist.com and replace with a blanket block.

    Flameeyes committed Aug 29, 2017
    As pointed out in Issue #15, gist.com doesn't exist any more. This removes
    the validation, and replaces it with a blanket block if someone tries to
    pass for it.
Commits on Aug 23, 2017
  1. Verify AppleBot according to Apple's specs.

    Flameeyes committed Aug 23, 2017
    Also allow AppleBot to go through despite badly passing for a browser.
  2. Fix obvious typo in the rule.

    Flameeyes committed Aug 23, 2017
  3. Validate BingPreview the same way as BingBot.

    Flameeyes committed Aug 23, 2017
    Also, skip all fake-browser tests for already-validated bots since they are pointless.
Commits on Aug 16, 2017
  1. Add epicbot to the list of bad robots.

    Flameeyes committed Aug 16, 2017
    This appears to be running a lot of HEAD requests, and fetch all new content. But it also provides https://www.google.com/ as the referer in all cases, which is clearly wrong.
Commits on Jul 26, 2017
  1. Add gitignore file.

    Flameeyes committed Jul 26, 2017
  2. Relicense the ruleset under Apache-2 license.

    Flameeyes committed Jul 26, 2017
    This has been signed off over Twitter by both Johann
    https://twitter.com/ercpe/status/890068271055216640 and Konstantin
    https://twitter.com/cryothresh/status/889962341911199744.
    
    Signed-off-by: Konstantin Pavlov <thresh@videolan.org>
    Signed-off-by: Johann Schmitz <johann@j-schmitz.net>
Commits on Jul 2, 2017
  1. Fix mistake in rule.

    Flameeyes committed Jul 2, 2017
  2. Block some vulnerability scanners.

    Flameeyes committed Jul 2, 2017
    These appear to be looking for database backups, so make sure to block all of them and blacklist them after it happened.
Commits on Jun 27, 2017
  1. Block requests with referrer spam coming from snacktory.

    Flameeyes committed Jun 27, 2017
    The library is not maintained and this appears to be some scraper that can't be bothered to fix it, so just filter the requests.
  2. Block SEMrushBot as a bad robot.

    Flameeyes committed Jun 27, 2017
    This SEO company misrepresents the bot in their page
    https://www.semrush.com/bot/ by saying that it's used for AdSense reports,
    but it is not actually related to AdSense.
    
    Block the bot and get rid of it.
Commits on Apr 22, 2017
  1. Also add Linguee to the list of bad bots.

    Flameeyes committed Apr 22, 2017
    They request more than one page per second.
  2. Ban MegaIndex as a bad robot.

    Flameeyes committed Apr 22, 2017
    They seem to be another SEO/marketing bots but they have NO ratelimiting,
    and keep requesting page without any regard for bandwidth usage, fetching
    one page per second.
Commits on Feb 14, 2017
  1. Fix typo in error message

    Flameeyes committed Feb 14, 2017
Commits on Dec 6, 2016
  1. Merge pull request #13 from thresheek/master

    Flameeyes committed Dec 6, 2016
    Fixed rules to load under modsecurity v3.
Commits on Dec 5, 2016
Commits on Nov 5, 2016
  1. Ignore the Mozilla/5.0 in the middle of LinkedInBot User-Agent.

    Flameeyes committed Nov 5, 2016
    Current example:
    
    User-Agent: LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
  2. Remove nbot from the list of bad bots because it matches LinkedInBot.

    Flameeyes committed Nov 5, 2016
    I also couldn't find a hit from this in the past two months so I assume it's just gone.
  3. Add a validation for LinkedInBot.

    Flameeyes committed Nov 5, 2016
    They started actually accepting gzip-encoded responses so it's time to make sure they are really them.
Commits on Apr 26, 2016
  1. Merge pull request #12 from hongaar/patch-1

    Flameeyes committed Apr 26, 2016
    Removed flavors.me
Commits on Apr 25, 2016
  1. Removed flavors.me

    hongaar committed Apr 25, 2016
    Fixes #11
Commits on Feb 22, 2016
  1. Merge pull request #10 from ercpe/ref-spam

    Flameeyes committed Feb 22, 2016
    More referrer spam, mostly russian
Commits on Feb 21, 2016
  1. More referrer spam, mostly russian

    ercpe committed Feb 21, 2016
Commits on Feb 8, 2016
  1. The recent russian referspammers fixed their script.

    Flameeyes committed Feb 8, 2016
    Now the general http matching is not working anymore, so blacklist them all explicitly.
Commits on Feb 6, 2016
  1. Remove cidr.bl.mcafee.com

    Flameeyes committed Feb 6, 2016
    This list now imports Spamhaus's Policy Blacklist which will block lots of valid IPs for browsers, but not for mailers.