adblock.mahakala.is has some issues

[kurumushi]

This hosts file blocks many legitimate sites, including both the download site for adobe flash and rival adblock lists (it blocks hosts-file.net which is one of your pihole sources.) It also includes lines at the top that give domain names to the hosts 'debian,' and 'android,' which may not be wanted by some people.

[jacobsalmela]

The mahakala list has caused some other problems, too, but it definitely helps more than it hinders. I show these entries at the top of my list:

localhost
localhost.localdomain
android
android.localhost
test
test.localhost
debian
debian.localhost

I agree these are probably not the best entries to include by default. You could easily add these to /etc/pihole/whitelist.txt but not everyone will do that.

I will look into a way to remove these entries from this, or any list.

[rmceoin]

In what situation would a single bare word be appropriate to blacklist? Unless we want to block an entire TLD. But do any of the downloaded lists contain a TLD? If not, then I'd say skip any line where the host specified is a single word without a period.

For the rest perhaps a regex to strip lines with *.localdomain and *.localhost

[jacobsalmela]

I'm not sure grep will interpret the *. Currently, we are using awk to append a $ to the end of each line. I have always struggled with regex, but I agree with your logic. Any one word lines probably isn't a domain.

We might be able to add all of those entries at the beginning of the user's whitelist.txt the way I used to do it for the hosts file. But I think some sort of regex/data validation would be better than hardcoding stuff.

[kurumushi]

Once the file is in hosts format, this command will delete all domains that do not contain a dot.

sed -e '/^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\} .*\..*$/!d'

[jacobsalmela]

Awesome! sed experts are the best. Would you mind explaining how each section of the command works?

[korhadris]

Hopefully I'm not stealing your thunder, or being overly descriptive below...

"/": sed delimiter
"^": beginning of line
"[0-9]\{1,3\}": any 1 to 3 digit number
"\.": a single '.' character
These are repeated 4 and 3 times to match an IPv4 address.
" ": a single space
".*\..*": anything.anything (anything could include other '.' characters too), so it'll match every name that at least has one '.' in it.
"$": end of line, if you want to match a '$' character, you should use "\$"
"/": sed delimiter
"!d": delete non matches

All that said, you could use "grep -e" to match the lines you are looking for as well. With the same regexp (possibly modified for $ matching if that's what you are looking for).

If it is a single word you are looking to remove you could also try something like:
grep -ve '^[0-9.]\+ \+\w\+$'

Additional regexp explanations:
"[0-9.]\+": match one or more ("\+" part) numbers or '.' characters
" \+": match one or more spaces (I don't know if you existing formatting only creates a single space, then you could skip the \+ part)
"\w\+": one or more "word" characters (letters and numbers)
"$": end of line, important in my example in particular, since this is looking for single names to match and not print, versus original regexp that looks for dotted names to match and not delete.

[kurumushi]

No need to worry about my thunder, I think you explained it nicely.

[gomoku]

I agree that mahakala list is agressive and blocks some reasonable websites wherein it would be better to only dispose the websites by blocking the agressive objects (like ads, popups, scripts, trackings, spys, cookies) and keep the websites working. It blocks many reasonable porn tubes (which aren't neither fake nor 3rd party based):

beeg.com
bonertube.com
bongacams.com
dirtydirtyangels.com
drtuber.com & www.drtuber.com
empflix.com & www.empflix.com
eporner.com
extremetube.com
fantasti.cc
heaven-portal.com
lighthouseangels.net 
motherless.com
nuvid.com & www.nuvid.com
pervclips.com
pichunter.com 
pornerbros.com 
pornoxo.com
porntube.com & www.porntube.com
redtube.com 
rude.com
smutty.com
spankbang.com
spankwire.com 
sunporno.com 
tnaflix.com 
txxx.com
www.cliphunter.com 
www.daftporn.com
www.myxvids.com
www.porn.com
www.pornicom.com
www.proporn.com
www.thenewporn.com
www.wankoz.com
xtube.com
youjizz.com & www.youjizz.com & pics.youjizz.com

I guess there will be more with time.

As well some popular regular websites:

1. https://www.playok.com/ (the domain www.playok.com ) - Hey, the people can't just play games!
2. http://en.chessbase.com/ (the domain chessbase.com) - a popular chess shop
3. https://www.skype.com/ (the domain skype.com) - a popular messenger program

as well it blocks also another big hosts file, as @kurumushi has already mentioned: http://hosts-file.net huh ?

Another thing that is strange is that there is no contact to mahakala author as mahakala list has no either support site like an issues tracker or a forum, nor any contact info in the list, like commented introduction section at the beginning of the list file, like for example in http://hostsfile.mine.nu/hosts0.txt:

##################################################################### 
# The Hosts File Project http://hostsfile.mine.nu 
# Global Advert Servers Blocklist - Personal Edition 
##################################################################### 
# Release 2012-05-07 
# Servers Verified as up and running 2012-05-07 (by dns resolving) 
# Updated sorted and maintained by Andrew Short (sh0rtie) 
# Contact: wduk10@hotmail.com 
##################################################################### 
# A big thank you to all contributers (too many to mention) 
# who really have made this project a success, well done :) 
# Licensed under the LGPL a copy of the license may be viewed at 
# http://www.gnu.org/licenses/lgpl.txt 

And the only two links which lead to mahakala I've found in Google are:
https://mahakala.is/ - here is nothing but a logo of meditating Mahakala god like Budda, I guess the point is he watches over us and protects internet users from the malicious websites
https://adblock.mahakala.is - the list's hosts file

As seen mahakala blocks so many reasonable websites, and there is no contact / bug tracker / forum to send the broken websites to be unblocked, thus the only way to fix is on the client's side by editing manually the list and removing the affected domains, or creating the exception rules in for ex. in a browser adblocker.

[WaLLy3K]

If you want to contact the Mahakala list maintainer, grab the gmail address that's listed when you whois the domain. The maintainer is very responsive to false positive queries :)

[gomoku]

Thanks, I've missed that :) I've send an email to the author right now:

Title: Blocked websites
Hello, could you refer to?:
#35 (comment)
You may refer here or on githib as well.
Thanks.

However, you agree that it is a little hidden, and for most people which are non-tech would be better to have more easier & public contact like an issues tracker or a forum, to not only send broken websites, but also for a dicusssion between people. I suppose the author could hide the contact to avoid publically replying to awkward/uncomfrotable questions related to blocked websites.

[mahakala]

hi guys,

i'm the hoster of this file on mahakala.is

for issues look at https://forum.xda-developers.com/showthread.php?t=1916098 - thats the origin of the file.

i could fork it and do the requested changes discussed here - but to be honest, i'd like to keep my work as simple as possible.

all the best

[gomoku]

From https://forum.xda-developers.com/showthread.php?t=1916098:

because of the extreme blocking nature.

Oh.

[dschaper]

@mahakala Thanks for posting here with us!