Customising Sources for Ad Lists

W0CHP edited this page Apr 4, 2018 · 7 revisions

Pi-hole's Default Block Lists

UPDATE: As of version 3.0, adlist management can now be done via the web UI, you no longer need to touch the command line for this! You can find it under Settings > Pi-Hole's Block Lists.

By default, when pihole -g pulls in lists of domains to block, we combine several lists, which are defined in /etc/pihole/adlists.list:

# The below list amalgamates several lists we used previously.
# See `https://github.com/StevenBlack/hosts` for details
##StevenBlack's list
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

##MalwareDomains
https://mirror1.malwaredomains.com/files/justdomains

##Cameleon
http://sysctl.org/cameleon/hosts

##Zeustracker
https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist

##Disconnect.me Tracking
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt

##Disconnect.me Ads
https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt

##Hosts-file.net
https://hosts-file.net/ad_servers.txt

How To Parse A List To Get Just The Domain

Imagine you found a list you want to use, but it is formatted with a bunch of extra characters:

||unlimited-hacks.net^
||pakcircles.com^
||cracksplay.com^
||fbgamecheatz.info^
||linkz.it^

You can use sed and/or awk (or other commands) to remove the extra characters to get just the domain name. It helps to be familiar with scripting, but if you wanted to parse down the list above, you could do something like this:

curl -s http://some.list | sed 's/^||//'

This would remove the two pipes at the beginning of the lines, so your list would then look like this:

unlimited-hacks.net^
pakcircles.com^
cracksplay.com^
fbgamecheatz.info^
linkz.it^

Then, you could use sed again, or even something like cut. Since the domains won't have a carat in the name, you can use it as a delimiter with the cut command to display only the domain name.

curl -s http://some.list | sed 's/^||//' | cut -d'^' -f-1

Which leaves you with just the domain names:

unlimited-hacks.net
pakcircles.com
cracksplay.com
fbgamecheatz.info
linkz.it

There is more than one way to parse the list down and there is no right way, however, some methods are faster. If you can combine most of your parsing into a single awk command, it can process a large list much faster. For each | that you use in the command, you are slowing down the processing as it is running in another subshell.