adkilla

Highly configurable adblocker, also useful against malware and phishing.

Highlights

• Blacklist and whitelist can be edited to choose lists to download and use as well as individual domain/hostnames.
• Current configuration merges 55 well-selected ad, tracking, malware, phishing and telemetry site lists and one whitelist.
• Can read a multitude of adblocker list formats.
• Over 1.6 million hosts/domains blocked as of January 2021, with almost no false-positives!
• When using recursive (subdomain) blocking, the number of domains decreases to ~280k, massively reducing memory usage.
• Can export into DNS server configuration file format: Unbound (implemented) BIND and others (on request).

Recommended usage: From crontab, about monthly update
Recommended parameters:

adkilla -v --downloadstripmerge --unbound -r=127.0.0.1 -d=/var/adkilla -s -t=/var/unbound/unbound_include.txt -c=/usr/local/etc

Parameter explanation:
-v : more verbose output
--downloadstripmerge : download, strip and merge the black+whitelists
--unbound : create unbound include file in target directory
[the following options need only to be set if not using the default values shown below]
-r=127.0.0.1 : set the IP to redirect to for blocked domains/hosts
-d=/var/adkilla : use this directory as storage for downloaded and processed files
-s : this option turns on blocking of subdomains of blacklisted domains
-t=/var/unbound/unbound_include.txt : target path/filename for the unbound include file
-c=/usr/local/etc : where the config files, i.e. black- and whitelists are.

To see all available options, run adkilla --help

Please read the config files for more usage instructions:
adkilla_blacklist.conf : blacklist file
adkilla_whitelist.conf : whitelist file

Do not forget to create the directories if they aren't there yet!
If you are not running the script as root, the working and target directories must be set so the user can read and write!
Runtime dependencies: Perl, wget

How it works

There are four passes:

• Download
• Strip
• Merge
• Generate DNS include file for Unbound (and maybe others, on request)

Download

Every time this step is called, a subdirectory in the work directory is created, with the name "YYYY-MM-DD-HH:MM:SS".
Thus you can keep older downloaded versions for archive etc.
Space usage is low compared with modern storage sizes, about 100MB for all downloaded and resulting processed files. If you are concerned about space usage, install a packer or cleaner script.
The chronologically as alphabetically latest directory created automatically becomes the work subdirectory for the following steps, too.
The files are downloaded with their names consisting of their URL (with slashes and spaces replaced by an underscore), prefixed by "black" or "white", respective what kind the list is.

Strip

As the blocklist files are in numerous formats, some even tarpacked, they have to be preprocessed.
HTML, different ad blocker list formats, malformed list entries, other errors in lists and all this.
The strip pass removes (almost) all this garbage, leaving only the domain/host names that are of interest to us.
The result files have the same name as the originals, with added filename extension ".stripped".

Merge

This pass merges the (in many cases redundant) information from the blocklists, so there are no double entries in the resulting final blocklist.
Depending on whether you want recursive subdomain blocking (-s option) or normal non-recursive blocking, the resulting file size and memory usage differ considerably.
As there is no need to store subdomains of already-blocked domains, the memory usage is considerably less when using the -s option.
Using the current blocklists (Jan 2021), peak memory usage is <400MB compared to <1.4GB when using normal, non-recursive blocking.
The result files from the merge pass are:
blackmergefile.txt and whitemergefile.txt: These files contain the merged black and whitelists, respective.
finalmergefile.txt: This file contains the final blacklist, consisting of the merged blacklist minus the merged whitelist.

Generate DNS include file for Unbound (and maybe others, on request)

The final mergefile only consists hosts/domain names and this needs to be blown up so Unbound can use it.
The way the resulting "unbound_include.txt" file gets blown up depends on your setting, default normal, or with -s option recursive, including subdomains.
This file then gets copied into /var/unbound, if you specified no other target.

And now you can reload unbounds' configuration and enjoy a less annoying internet :)