Referer Spam Blacklist
This is a frequently updated list of referer spam domains.
This repository is a fork of Piwik's referrer spam blacklist, with the following main differences:
- Domains are added more frequently to keep up with the spammers (Piwik's list requires a vote and a pull request per added domain, which slows down the update, and prevents automatic updating). Piwik's list changes are also merged back regularly in this repository.
- Large lists of confirmed referer spam domains from other contributors or projects are also merged back
- Dead domains are regularly removed from the list
As a result this list is more than 5 times larger (as of this writing) than Piwik's list, while having less dead domains.
The list is stored in this repository in
spammers.txt, with one host per line, and Unix line endings.
Parsing the file should be pretty easy using your favorite language.
Sub-string matching on domain names can be done with this list, so
semalt.com is enough to block all subdomain referers too, such as
However, there are cases where you'd only want to add a subdomain but not the root domain. For example, add
refererspammer.tumblr.com but not
tumblr.com, otherwise all
*.tumblr.com sites would be affected.
On the contrary, if you get spam with referer
www.domain.com, you can safely assume that all
domain.com subdomains are spam.
To make duplicate detection, and merges easier, the list is sorted with Unix's sort, with the command:
sort -uf -o spammers.txt spammers.txt
Usage with Fail2ban
This list can be used with Fail2ban.
A script is provided to automatically generate or update a filter (located in
/etc/fail2ban/filter.d/apache-referer.local) that will watch server logs, and automatically ban IPs that send HTTP requests with a domain in the blacklist as referer. It has been tested with Apache, but should work with any webserver configured to log using the standard combined log format, provided you change log locations in the script.
It is recommended to run the script at least every week with cron, to keep the list up to date:
curl https://raw.githubusercontent.com/desbma/referer-spam-domains-blacklist/master/update-fail2ban-referer-filter > /etc/cron.weekly/update-fail2ban-referer-filter chmod +x /etc/cron.weekly/update-fail2ban-referer-filter
You also need to edit
/etc/fail2ban/jail.local, to locate the Apache logs, and configure ban time, eg.:
[apache-referer] enabled = true maxretry = 1 # 90 days bantime = 7776000 port = http,https filter = apache-referer logpath = /var/log/apache*/*access.log
Then, run the script a first time to generate the filter:
Usage with Google Analytics
See this repository for a tool to generate and update a Google Analytics filter from this list to remove referer spam.
Although the list is mostly updated by automated scripts, contributions to the list are welcome.
To add domain(s) either open an issue or submit a pull request. The latter is prefered for large additions. Either way, please explain how you have found the referer domain(s) (it is important to avoid false positives).
If you think a hostname was added but is not actually a spam domain, please open an issue.