Reduce number of false positives when gravity unable to import domains - change the wording of the output

[PromoFaux]

What does this PR aim to accomplish?:

Currently when we analyse an adlist, we print out a warning if any invalid domains are detected. However, it is not always strictly true that there is something wrong with the entry - it is more that they are incompatible with Pi-hole

https://discourse.pi-hole.net/t/fresh-install-steven-black-list-invalid-domains/60523

This PR introduces a small list of common localhost domains (as included at the top of StevenBlack's list to discount these as being counted as "invalid", and also changes the wording of invalid domains to unusable entries.

Example output before these changes:

[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 168834 domains, 15 domains invalid!
      Sample of invalid domains:
      - 0.0.0.0
      - broadcasthost
      - fe
      - ff
  [i] List stayed unchanged

[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
  [i] Analyzed 168834 domains, 15 domains invalid!
      Sample of invalid domains:
      - 0.0.0.0
      - broadcasthost
      - fe
      - ff
  [i] List stayed unchanged

Example output after these changes:

  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
  [i] Imported 168819 domains, ignoring 2 non-domains
      Sample of non-domain entries:
      - fe
      - ff

  [i] Target: https://raw.githubusercontent.com/jankytay/pihole-troubleshooting/master/issue5008.txt
  [✓] Status: Retrieval successful
  [i] Imported 3 domains, ignoring 5 non-domains
      Sample of non-domain entries:
      - (\.|^)example6\.com$
      - (^|\.)example5\.com
      - (^|\.)example8\.com;reply=none
      - *.example3.com
      - ^example77?

The copy could probably be further clarified to mention that this is not an error - just for information, and that nothing is wrong, but it is late and I'm struggling to find the words

---

By submitting this pull request, I confirm the following:

1. I have read and understood the contributors guide, as well as this entire template. I understand which branch to base my commits and Pull Requests against.
2. I have commented my proposed changes within the code and I have tested my changes.
3. I am willing to help maintain this change if there are issues with it later.
4. It is compatible with the EUPL 1.2 license
5. I have squashed any insignificant commits. (git rebase)

---

• I have read the above and my PR is ready for review. Check this box to confirm

[pralor-bot]

This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/fresh-install-steven-black-list-invalid-domains/60523/27

[PromoFaux]

@jfb-pihole can I have your thoughts on this version, too, please?

[jfb-pihole]

"Sample of unusable entries:"

I would change this to "sample of entries not imported into Pi-hole"

[rdwebdesign]

The expression "unusable entries" tries to make it clear Pi-hole can't use those entries...

I know the difference is very small (if any), but I think the expression "not imported into Pi-hole" will raise a lot of questions like "Why Pi-hole didn't import everything? What is wrong?".

We can explain better using something like "X lines without domains not imported by Pi-hole", but this looks too long.

[jfb-pihole]

I'm OK with that change. We just need to indicate that the domains weren't imported.

[PromoFaux]

  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
  [i] Imported 168819 domains, ignoring 2 non-domains
      Sample of non-domain entries:
      - fe
      - ff

  [i] Target: https://raw.githubusercontent.com/jankytay/pihole-troubleshooting/master/issue5008.txt
  [✓] Status: Retrieval successful
  [i] Imported 3 domains, ignoring 5 non-domains
      Sample of non-domain entries:
      - (\.|^)example6\.com$
      - (^|\.)example5\.com
      - (^|\.)example8\.com;reply=none
      - *.example3.com
      - ^example77?

[pralor-bot]

This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/stevenblack-adlists-show-15-invalid-domains/60641/2

[pralor-bot]

This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/pi-hole-web-v5-18-2-and-core-v5-15-1-released/60695/1

[thomasmerz]

Great PR! I love it. Because now it's not "something wrong" anymore, but only "an info in a footnote" 😆

[dschaper]

These changes will lead to a high system load when fed a list that contains nothing of value or use. Any list that is in ABP format will try to filter out the common localhosts from a long string of invalid information (read ABP regex) and Pi-hole will try its hardest to salvage the list.

Looks like we need to have an intermediate step that looks for any hosts entry in the supplied list and if there are none then stop. I thought we had ABP filtering before but that may have been to try to convert ABP in to hosts format. If I can find that detection regex then we can reuse it to just remove the invalid entries and then count the length of the array of hosts entries. If there are none then we don't need to proceed. Sample list that doesn't work for Pi-hole anymore is https://big.oisd.nl. The older https://dbl.oisd.nl has been removed and we safely note that the list is gone. But people will try to replace dbl with big and we shouldn't crash just because they are feeding garbage to Pi-hole now.

[yubiuser]

Looks like we need to have an intermediate step that looks for any hosts entry in the supplied list and if there are none then stop.

If you want to go for an all-or-nothing approach, just check the number of lines of ${target} and if it's 0 skip the removal of false positive.

[dschaper]

Okay, ideally we should only remove false positives from potential positives but I haven't looked too deeply at the code to see how it all is being processed now. (sed and regex is such a nightmare in bash...)

[yubiuser]

The false_positive is misleading here. They are false_positive of being "unusable". We first get all non-domains and remove the false_positive from this list leaving the real "unusable". Therefore we first need all the garbage to sort out a few items.