[Feature] Implementation of a way to disable the SPECIAL rules

[dnmTX]

Describe the bug
Ultimate-Hosts-Blacklist/dev-center#16

[funilrys]

Closing for now 😹

[funilrys]

Let's reopen this then 👍

[dnmTX]

We'll need some exception rule for this one as well.
wpad
wpad.corp
wpad.example
wpad.home
wpad.lan
wpad.local
wpad.localdomain
wpad.mail
wpad.test

[funilrys]

Feature introduced with 7a5b64f.

Please DO NOT close this issue manually. This issue is going to be closed automatically once 7a5b64f will be merged to master.

[funilrys]

By default, we test for public accessible domains @dnmTX. Those domains can be tested as --local without any issue but not on public usage. They will always be set as INACTIVE or INVALID until I exclude them for every test explicitly.

After reading https://pi-hole.net/2018/09/10/mitigate-a-new-cert-vulnerability-598349-with-an-entry-in-etc-hosts/ I find it hard to exclude it for any test as every network may have its own local domain.

Between I already handle part of the problem as all domain which ends or is exactly in one of the following are ignored when testing a file:

localhost
localdomain
local
broadcasthost
0.0.0.0
allhosts
allnodes
allrouters
localnet
loopback
mcastprefix
ip6-mcastprefix
ip6-localhost
ip6-loopback
ip6-allnodes
ip6-allrouters
ip6-localnet

I'll not handle it @dnmTX. I prefer to handle a list of common local domains which can be found on the wild instead of setting a permissive rule which may exclude everything which starts with wpad. That's not what we want.

[dnmTX]

@funilrys about the wpad.* that it's perfectly fine.It's more like custom for now so whoever want it can add them as a addition(that's what i did on my end).
Priority(can't really stress enough when i mention PRIORITY) here is the *.doubleclick.net domains and anything else that YOU and MITCH decided that should be considered INACTIVE but in reality is very much ACTIVE and dangerous.It's one thing to discuss/enhance the whitelist which is optional and another when me and others(i guess) who are using the clean.list as a main source thinking that we getting the same protection as the original source and there is no way to add those domains because there is no choice.
Sorry for being so critical but this is important to me.
Please,anything else you can think of about active domains that were considered inactive for really not good reason,include them here in those special rules.
THANK YOU !!!!!

P.S. Just as a example you look at the INACTIVE folder in @lightswitch05's lists and you can see how many are there and they are all very much ACTIVE/LIVE 😞

[funilrys]

@dnmTX This tool was and is developed independently. @Ultimate-Hosts-Blacklist use PyFunceble which his own configuration and @dead-hosts use PyFunceble which its default configuration + what is needed to run under Travis CI.

The SPECIAL rules are there and will stay there. Also, the SPECIAL rules are not only about taking down domains. For example, if I disable that all IP ranges will be set as inactive which is not great for the DNS server or user which take the IP range and converts them into a list of IP.

The SPECIAL rules where developed because we did a constatation and we tested our constatation. Also, we were and are pretty sure that if the status code changes PyFunceble will put them into the ACTIVE list automatically on next retest. We did our test and we don't only look at the domain from outside. With @mitchellkrogza we took such decision not because we wanted that. Our repositories and list are used around the globe we can't consider taking such decision by design just because "we don't want them".

We had over 70+ domains which could be into that SPECIAL rules for INACTIVE ==> ACTIVE and about 50+ for INACTIVE ==> ACTIVE but just because those domains were working differently we choose not to include them.

If you want a list with PyFunceble and only NSLOOKUP without WHOIS or/and the SPECIAL RULES (that's actually your expectation of the clean.list) you should consider running your own tests as those will take something like 4 hours if not less if you exclude the big one for the whole @Ultimate-Hosts-Blacklist. For a list of 3000 elements without WHOIS, it will take less than 5 minutes if not more depending on your DNS server and connection speed.

I will update the graphs in the coming versions (added to my backlog) in order to be and stay more transparent even if it was already well documented.

Thanks for the critic but on this, the option is available now on the dev version. The SPECIAL rules will stay for now. I may disable it into @Ultimate-Hosts-Blacklist after consulting @mitchellkrogza but just not in the coming hours.
For me, the most important now is to focus on building and improving the current infrastructure or build and work on top of an infrastructure which with less cost will free us for the restriction of Travis CI (some lists take a really long time) for both @dead-hosts and @Ultimate-Hosts-Blacklist and which will let us do more, for example, proposing an extra file/list without the SPECIAL rules as you implicitly requested.

I respect your opinion but sometime I can't follow every opinion.

Cheers,
Nissar

[dnmTX]

Well...let me know what you guys came up with,especially about the *.doubleclick.net domains as they are the most dominated once in various lists followed by *.wordpress.com and *.tumblr.com=>mostly present in malware lists.So i know what to do next.
There is no point to argue,it's obvious to me that we see things differently on the subject here.