adblock plugin

[ffries]

Dear Friends,

Lede offers this marvelous plugin:
https://github.com/openwrt/packages/tree/lede-17.01/net/adblock/files

It works very well and wipes all adds without breaking applications.

The adblock plugin relies on a single bash script:
https://github.com/openwrt/packages/blob/lede-17.01/net/adblock/files/adblock.sh

It is compatible with dnsmasq and unound and relies on "lying DNS".

I am aware that it is possible to integrate DNSBLs in a proxy, but here the solution is lean and simple, so I wonder whether this script could be distributed and adapted for opnsense.

Kind regards,
Frenc Fries

[fabianfrz]

This does already work with OpenDNS, the Web Proxy (https://docs.opnsense.org/manual/how-tos/proxywebfilter.html), ICAP and many other ways. There is also a script working with OPNsense somewhere around.

[ffries]

This question is for development purpose, as I may be interesting in stepping in.

I understand that proxies already support banlists. It is also possible to create a list of hosts and implement it at packet filtering level (pf).

I was wondering of a DNS alone approach (lying DNS), as it proves to works pretty with little computing burden. This is how it is done in LEDE (OpenWRT reboot). At present (2017), all adds are pointing to full-qualified hostnames. None of them are pointing to IPs either v4 or v6. This is why the DNS approach seems well enough. The script works in conjunction with dnsmasq or unbound.

I might be wrong, but I think this is the right approach until adds have publics IPs and not FQDN.

Furthermore, when you think of proxy, you need to use a transparent-reverse one to make sure that all traffic actually goes through the proxy. With lying DNS, you can make sure that no-add, whatever using direct or proxy connection, goes through.

Please make any comment that might save my future development time ...

[ffries]

I am catching up:
OpenDNS is avaiable from services_opendns.php

Unfortunately, a private company will not want to leak its DNS.
Not today in 2017, it should be done inside the firewall.

But this basically what I am looking for, so I will enquire more.

Any comments are welcome.

[fabianfrz]

I found the script working on OPNsense:
https://devinstechblog.com/block-ads-with-dns-in-opnsense/

[ffries]

Indeed, adblock.sh could be easily ported to OPNsense.

banlist can be fetched from:
https://raw.githubusercontent.com/openwrt/packages/lede-17.01/net/adblock/files/adblock.conf

i will try fist to implement it as an IP banlist to see if my firewall 'survives' 100.000 entries.

[ffries]

Most blacklist need to be processed using a regex.

Example from:
https://raw.githubusercontent.com/openwrt/packages/lede-17.01/net/adblock/files/adblock.conf

'securemecca'
http://securemecca.com/Downloads/hosts.txt'
'$0 ~/^127.0.0.1[ \t]+([A-Za-z0-9_-]+.){1,}[A-Za-z]+/{print tolower($2)}'

I am opening a separate ticket on opnsense core.

[fabianfrz]

there is no regular expression required.
the lines with comments can be removed using grep -v '#', the useful lines can be extracted using grep '127.0.0.1' and the hostname should be possible to be extracted using cut -w -f2.

So theoretically this would work without a regular expression.

[ffries]

Sounds very easy, you are right.
The idea is at some point we need a banlist downloaded in OPNsense, either in core or in a plugin.

I opened this issue to ask core team what they are waiting for:
opnsense/core#1836

Then, i can start my own dev.

[phedoreanu]

The original link seems to be down, but I managed to grab a copy of the update-hosts.sh:

https://gist.github.com/phedoreanu/0f28594b17caa7cb6da0ef235d16a1f4

[Woi]

@ffries Not sure if it's of any use for this issue, but a FYI can't hurt: I yesterday stumbled upon https://pi-hole.net/. It seems like they use a DNS-based approach, along with a (optional?) web server for statistics, graphs and some settings.
There was already some discussion about porting it to BSD: https://discourse.pi-hole.net/t/freebsd-compatability/2092
The code is on github: https://github.com/pi-hole

[rhysxevans]

Ok, not the best but have done the following to get something basci working for me. Obviously work needs to be done on reporting and putting config into the gui (I'm not a developer so cant help out there). But was thinking the lists in the script could be bought into the gui as options.

Via CLI
#Get fetcher script
curl -sS -L https://raw.githubusercontent.com/Aracktus/DNS-Unbound-Blocklist-Downloader/master/dns-unbound-blocklist-downloader.py > /var/unbound/dns-unbound-blocklist-downloader.py

#Update some of the variables to reflect the OPNSENSE locations
sed -i -- 's/\/etc\/unbound\/'\/var\/unbound\/conf\.d\//g' /var/unbound/dns-unbound-blocklist-downloader.py
sed -i -- 's/local-blocking-data\.conf/99-local-blocking-data\.conf/g' /var/unbound/dns-unbound-blocklist-downloader.py

#how to restart unbound in opnsense??

sed -i -- 's/\/usr\/sbin\/service unbound restart/\/usr\/local\/sbin\/unbound-control -c \/var\/unbound\/unbound\.conf reload/g' /var/unbound/dns-unbound-blocklist-downloader.py

#The next section is manual
Remove the "AdAway mobile ads" section as this is now dead by the look of things
You can add sources as required

#Download Files
/usr/local/bin/python2.7 /var/unbound/dns-unbound-blocklist-downloader.py

#Set permissions, not 100% sure this is actually required, and could add it to the actual script if it is
chown unbound:unbound /var/unbound/conf.d/99-local-blocking-data.conf

#These Seds not required if removing "Someone Who Cares" entry (comments cause issues)
sed -i -- '/http\:\/\//d' /var/unbound/conf.d/99-local-blocking-data.conf
sed -i -- '/Site A/d' /var/unbound/conf.d/99-local-blocking-data.conf

#Add to the advanced section of unbound in GUI (Save and reload)
include: /var/unbound/conf.d/99-local-blocking-data.conf

#Setup Cron Job via gui, once daily
/usr/local/bin/python2.7 /var/unbound/dns-unbound-blocklist-downloader.py

The above may be usefull or not, and I am sure there are more efficient ways to do this, thanks

[rhysxevans]

jumped the gun on the cron job via the gui, had to do the following

create the file "/root/dns-unbound-blocklist-downloader.sh" and add the below to it

#!/bin/sh
/usr/local/bin/python2.7 /var/unbound/dns-unbound-blocklist-downloader.py
exit 0

Mark the script as executable

chmod +x /root/dns-unbound-blocklist-downloader.sh

create the file "/usr/local/opnsense/service/conf/actions.d/actions_dnsblacklistupdates.conf"

put the following into it

[start]
command:/root/dns-unbound-blocklist-downloader.sh
parameters:
type:script
description:Update DNS Blacklists
message:Update DNS Blacklists

Restart configd

service configd restart

At this point you should be able to go into the GUI to set the daily cron job

[mimugmail]

We might close this one too, we have BIND plugin with DNSBL and soon support within Unbound #808