Implement a default Referrer spam blacklist

[mattab]

it looks like ###.com are referer spamming us. Time to add a referer blacklist...

http://demo.piwik.org/index.php?module=CoreHome&action=index&idSite=7&period=day&date=yesterday#module=Referers&action=getSearchEnginesAndKeywords&idSite=7&period=day&date=yesterday

[robocoder]

I looked at the logs, and this appears to be a result of http://forum.piwik.org/read.php?2,74422. I've changed the 'X' to '#'. Hopefully, Google refreshes its cache soon.

[mattab]

I did the search but didn't see piwik forums in the results, hence why I thought this was spam - did you see the piwik forums in the results?. This is funny hehe

[mattab]

ok it appears it's not spam indeed (don't look at what are the keywords for today that's just wrong...)

[robocoder]

LOL

[mattab]

Reopening, I found this list of referrer spam which we could reuse... when this becomes a problem.

[anonymous-matomo-user]

see also
http://forum.piwik.org/read.php?2,75066,75102

short summary:
a lot of fake visits (http://en.wikipedia.org/wiki/Referrer_spam) making the statistics almost unusable (we do not publish our stats. Nonetheless, internal "reading" of the webstats is now a pain in the heck due to this many false data).

Solution would be: allow a admin to create/update a black list (maybe a regex list) for referrer spam sites, which are simply ignored (not counted at all) by piwik. Similar to the IP blacklist.

[anonymous-matomo-user]

I get a lot of referrer spam too. As I have very little visitors, this spammers makes for round about 90% of the unique visitors. Beside visitors coming from Facebook.com filtering all referrers that do not contain any url path, query or fragment would do the trick in my case.

Having a central repository would be a nice solution, even if that could be abused. Maybe this could be a solution that could be implemented by some piwik hosters.

[mattab]

Looks like the guys semalt.com are hard at spamming the world. we should really blacklist all visits on Piwik from this domain. And put the list in the config file so we can add new websites to block later.

[mattab]

semalt is acting up again and causing user frustration eg. in the forums. Let's kill it!

[mattab]

In b1fe857: Fixes #2268 Implement a default Referrer spam blacklist to block semalt.com spammer. (can be extended to other spammers on request!)

; All Visits with a Referrer URL host set to one of these will be excluded.
; If you find new spam entries in Referrers>Websites, please report them here: #2268
; List of known Referrer Spammers, ie. bot visits that set a fake Referrer field:
referrer_urls_spam = "semalt.com"

[anonymous-matomo-user]

Sites we track are being spammed by

semalt.com
www.semalt.com
semalt.com/crawler.php
www.semalt.com/crawler.php

[mattab]

Good, all these visits will now be excluded!

[anonymous-matomo-user]

I would like to point out that neither the forums, nor the issue tracker software, nor just about any other publicly accessible website allow lists of spam sites to be posted.

[mattab]

Please write them without http:// in front, or put them on pastebin.com and put the link here

[anonymous-matomo-user]

I get the following message when trying to post here:

Submission rejected as potential spam (Akismet says content is spam, Content contained these blacklisted patterns: ...)

Here is a pastebin of the last week:

http://pastebin.com/mpqCp5rZ