Fix edge-case where an adlist domain is blocked

[DL6ER]

What does this implement/fix?

Related discussion was going on on Discourse here.

This branch was forgotten about but still applies cleanly on development-v6 - it fixes the edge-case handling of an adlist domain being blocked by Pi-hole and - if so - using an external resolver (to be precise, the first of dns.upstreams) instead.

---

Related issue or feature (if applicable): N/A

Pull request in docs with documentation (if applicable): N/A

---

By submitting this pull request, I confirm the following:

1. I have read and understood the contributors guide, as well as this entire template. I understand which branch to base my commits and Pull Requests against.
2. I have commented my proposed changes within the code.
3. I am willing to help maintain this change if there are issues with it later.
4. It is compatible with the EUPL 1.2 license
5. I have squashed any insignificant commits. (git rebase)

Checklist:

• The code change is tested and works locally.
• I based my code and PRs against the repositories developmental branch.
• I signed off all commits. Pi-hole enforces the DCO for all contributions
• I signed all my commits. Pi-hole requires signatures to verify authorship
• I have read the above and my PR is ready for review.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[PromoFaux]

Tested on a container with ``

Added raw.githubusercontent.com to deny list and ran gravity:

[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✗] Status: Connection Refused
  [✗] List download failed: no cached list available

bash -x of gravity

a7aa1ce59b0b:/etc/pihole# bash -x /opt/pihole/gravity.sh 
+ export LC_ALL=C
+ LC_ALL=C
+ PI_HOLE_SCRIPT_DIR=/opt/pihole
+ utilsfile=/opt/pihole/utils.sh
+ . /opt/pihole/utils.sh
+ coltable=/opt/pihole/COL_TABLE
+ . /opt/pihole/COL_TABLE
++ '[' -t 1 ']'
+++ tput colors
++ '[' 8 -ge 8 ']'
++ COL_BOLD=''
++ COL_ULINE=''
++ COL_NC=''
++ COL_GRAY=''
++ COL_RED=''
++ COL_GREEN=''
++ COL_YELLOW=''
++ COL_BLUE=''
++ COL_PURPLE=''
++ COL_CYAN=''
++ COL_WHITE=''
++ COL_BLACK=''
++ COL_LIGHT_BLUE=''
++ COL_LIGHT_GREEN=''
++ COL_LIGHT_CYAN=''
++ COL_LIGHT_RED=''
++ COL_URG_RED=''
++ COL_LIGHT_PURPLE=''
++ COL_BROWN=''
++ COL_LIGHT_GRAY=''
++ COL_DARK_GRAY=''
++ TICK='[✓]'
++ CROSS='[✗]'
++ INFO='[i]'
++ QST='[?]'
++ DONE=' done!'
++ OVER='\r'
+ . /etc/.pihole/advanced/Scripts/database_migration/gravity-db.sh
++ readonly scriptPath=/etc/.pihole/advanced/Scripts/database_migration/gravity
++ scriptPath=/etc/.pihole/advanced/Scripts/database_migration/gravity
+ basename=pihole
+ PIHOLE_COMMAND=/usr/local/bin/pihole
+ piholeDir=/etc/pihole
+ whitelistFile=/etc/pihole/whitelist.txt
+ blacklistFile=/etc/pihole/blacklist.txt
+ regexFile=/etc/pihole/regex.list
+ adListFile=/etc/pihole/adlists.list
+ localList=/etc/pihole/local.list
+ VPNList=/etc/openvpn/ipp.txt
+ piholeGitDir=/etc/.pihole
++ getFTLConfigValue files.gravity
++ pihole-FTL --config -q files.gravity
+ GRAVITYDB=/etc/pihole/gravity.db
+ gravityDBschema=/etc/.pihole/advanced/Templates/gravity.db.sql
+ gravityDBcopy=/etc/.pihole/advanced/Templates/gravity_copy.sql
+ domainsExtension=domains
+ curl_connect_timeout=10
+ : /tmp
+ '[' '!' -d /tmp ']'
+ '[' '!' -w /tmp ']'
+ gravityDBfile=/etc/pihole/gravity.db
+ gravityDBfile_default=/etc/pihole/gravity.db
+ gravityTEMPfile=/etc/pihole/gravity.db_temp
++ dirname -- /etc/pihole/gravity.db
+ gravityDIR=/etc/pihole
+ gravityOLDfile=/etc/pihole/gravity_old.db
+ [[ -f /etc/pihole/gravity_old.db ]]
+ rm /etc/pihole/gravity_old.db
+ gravity_Trap
+ trap '{ echo -e "\\n\\n  ${INFO} ${COL_LIGHT_RED}User-abort detected${COL_NC}"; gravity_Cleanup "error"; }' INT
+ [[ '' == true ]]
+ [[ '' == true ]]
+ migrate_to_database
+ '[' '!' -e /etc/pihole/gravity.db ']'
+ upgrade_gravityDB /etc/pihole/gravity.db /etc/pihole
+ local database piholeDir auditFile version
+ database=/etc/pihole/gravity.db
+ piholeDir=/etc/pihole
+ auditFile=/etc/pihole/auditlog.list
++ pihole-FTL sqlite3 /etc/pihole/gravity.db 'SELECT "value" FROM "info" WHERE "property" = '\''version'\'';'
+ version=17
+ [[ 17 == \1 ]]
+ [[ 17 == \2 ]]
+ [[ 17 == \3 ]]
+ [[ 17 == \4 ]]
+ [[ 17 == \5 ]]
+ [[ 17 == \6 ]]
+ [[ 17 == \7 ]]
+ [[ 17 == \8 ]]
+ [[ 17 == \9 ]]
+ [[ 17 == \1\0 ]]
+ [[ 17 == \1\1 ]]
+ [[ 17 == \1\2 ]]
+ [[ 17 == \1\3 ]]
+ [[ 17 == \1\4 ]]
+ [[ 17 == \1\5 ]]
+ [[ 17 == \1\6 ]]
+ [[ '' == true ]]
+ gravity_CheckDNSResolutionAvailable
+ local lookupDomain=pi.hole
+ [[ ! -e /etc/pihole/local.list ]]
+ [[ -s /etc/pihole/local.list ]]
+ lookupDomain=raw.githubusercontent.com
+ timeout 4 getent hosts raw.githubusercontent.com
+ [[ -n '' ]]
+ timeout 4 dig +short raw.githubusercontent.com
+ [[ -n '' ]]
+ return 0
+ gravity_DownloadBlocklists
+ echo -e '  [i] Neutrino emissions detected...'
  [i] Neutrino emissions detected...
+ [[ /etc/pihole/gravity.db != \/\e\t\c\/\p\i\h\o\l\e\/\g\r\a\v\i\t\y\.\d\b ]]
+ mapfile -t sources
++ pihole-FTL sqlite3 /etc/pihole/gravity.db 'SELECT address FROM vw_adlist;'
+ mapfile -t sourceIDs
++ pihole-FTL sqlite3 /etc/pihole/gravity.db 'SELECT id FROM vw_adlist;'
+ mapfile -t sourceTypes
++ pihole-FTL sqlite3 /etc/pihole/gravity.db 'SELECT type FROM vw_adlist;'
+ mapfile -t sourceDomains
++ awk -F '[/:]' '{
      # Remove URL protocol & optional username:password@
      gsub(/(.*:\/\/|.*:.*@)/, "", $0)
      if(length($1)>0){print $1}
      else {print "local"}
    }'
+++ printf '%s\n' https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
+ local 'str=Pulling blocklist source list into range'
+ echo -e '\r  [✓] Pulling blocklist source list into range'
  [✓] Pulling blocklist source list into range
+ [[ -z https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts ]]
+ [[ -z raw.githubusercontent.com ]]
+ local url domain str target compression adlist_type
+ echo ''

+ str='Preparing new gravity database'
+ echo -ne '  [i] Preparing new gravity database...'
  [i] Preparing new gravity database...+ rm /etc/pihole/gravity.db_temp
+ output='++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp'
+ status=0
+ [[ 0 -ne 0 ]]
+ echo -e '\r  [✓] Preparing new gravity database'
  [✓] Preparing new gravity database
+ str='Creating new gravity databases'
+ echo -ne '  [i] Creating new gravity databases...'
  [i] Creating new gravity databases...++ cat /etc/.pihole/advanced/Templates/gravity_copy.sql
+ copyGravity='.timeout 30000

ATTACH DATABASE '\''/etc/pihole/gravity.db'\'' AS OLD;

BEGIN TRANSACTION;

DROP TRIGGER tr_domainlist_add;
DROP TRIGGER tr_client_add;
DROP TRIGGER tr_adlist_add;

INSERT OR REPLACE INTO "group" SELECT * FROM OLD."group";
INSERT OR REPLACE INTO domain_audit SELECT * FROM OLD.domain_audit;

INSERT OR REPLACE INTO domainlist SELECT * FROM OLD.domainlist;
DELETE FROM OLD.domainlist_by_group WHERE domainlist_id NOT IN (SELECT id FROM OLD.domainlist);
INSERT OR REPLACE INTO domainlist_by_group SELECT * FROM OLD.domainlist_by_group;

INSERT OR REPLACE INTO adlist SELECT * FROM OLD.adlist;
DELETE FROM OLD.adlist_by_group WHERE adlist_id NOT IN (SELECT id FROM OLD.adlist);
INSERT OR REPLACE INTO adlist_by_group SELECT * FROM OLD.adlist_by_group;

INSERT OR REPLACE INTO client SELECT * FROM OLD.client;
DELETE FROM OLD.client_by_group WHERE client_id NOT IN (SELECT id FROM OLD.client);
INSERT OR REPLACE INTO client_by_group SELECT * FROM OLD.client_by_group;


CREATE TRIGGER tr_domainlist_add AFTER INSERT ON domainlist
    BEGIN
      INSERT INTO domainlist_by_group (domainlist_id, group_id) VALUES (NEW.id, 0);
    END;

CREATE TRIGGER tr_client_add AFTER INSERT ON client
    BEGIN
      INSERT INTO client_by_group (client_id, group_id) VALUES (NEW.id, 0);
    END;

CREATE TRIGGER tr_adlist_add AFTER INSERT ON adlist
    BEGIN
      INSERT INTO adlist_by_group (adlist_id, group_id) VALUES (NEW.id, 0);
    END;


COMMIT;'
+ [[ /etc/pihole/gravity.db != \/\e\t\c\/\p\i\h\o\l\e\/\g\r\a\v\i\t\y\.\d\b ]]
+ output='++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp'
+ status=0
+ [[ 0 -ne 0 ]]
+ echo -e '\r  [✓] Creating new gravity databases'
  [✓] Creating new gravity databases
+ curl -V
+ grep -q 'Features:.* libz'
+ compression=--compressed
+ echo -e '  [i] Using libz compression\n'
  [i] Using libz compression

+ (( i = 0 ))
+ (( i < 1 ))
+ url=https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
+ domain=raw.githubusercontent.com
+ id=1
+ [[ 0 -eq 0 ]]
+ str=blocklist
+ adlist_type=gravity
+ saveLocation=/etc/pihole/list.1.raw.githubusercontent.com.domains
+ activeDomains[$i]=/etc/pihole/list.1.raw.githubusercontent.com.domains
+ echo -e '  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
+ local regex check_url
+ regex='[^a-zA-Z0-9:/?&%=~._()-;]'
++ sed -re 's#([^:/]*://)?([^/]+)@#\1\2#'
+ check_url=https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
+ [[ https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts =~ [^a-zA-Z0-9:/?&%=~._()-;] ]]
+ gravity_DownloadBlocklistFromUrl https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts 1 /etc/pihole/list.1.raw.githubusercontent.com.domains '' --compressed gravity raw.githubusercontent.com
+ local url=https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts adlistID=1 saveLocation=/etc/pihole/list.1.raw.githubusercontent.com.domains target= compression=--compressed gravity_type=gravity domain=raw.githubusercontent.com
+ local heisenbergCompensator= listCurlBuffer str httpCode success= ip cmd_ext
++ mktemp -p /tmp
+ listCurlBuffer=/tmp/tmp.doTNv9A8FU
+ mv /tmp/tmp.doTNv9A8FU /tmp/tmp.phgpb
+ listCurlBuffer=/tmp/tmp.phgpb
+ [[ -r /etc/pihole/list.1.raw.githubusercontent.com.domains ]]
+ str=Status:
+ echo -ne '  [i] Status: Pending...'
  [i] Status: Pending...+ blocked=false
+ [[ https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts != \f\i\l\e* ]]
+ [[ -n raw.githubusercontent.com ]]
+ case $(getFTLConfigValue dns.blocking.mode) in
++ getFTLConfigValue dns.blocking.mode
++ pihole-FTL --config -q dns.blocking.mode
++ dig raw.githubusercontent.com +short
++ grep 0.0.0.0 -c
+ [[ 0 -ge 1 ]]
+ [[ false == true ]]
++ curl --connect-timeout 10 -s -L --compressed -w '%{http_code}' https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts -o /tmp/tmp.phgpb
+ httpCode=000
+ case $url in
+ case "${httpCode}" in
+ echo -e '\r  [✗] Status: Connection Refused'
  [✗] Status: Connection Refused
+ local done=false
+ [[ '' == true ]]
+ [[ false != \t\r\u\e ]]
+ [[ -r /etc/pihole/list.1.raw.githubusercontent.com.domains ]]
+ echo -e '  [✗] List download failed: no cached list available'
  [✗] List download failed: no cached list available
+ database_adlist_number 1 0 0
+ gravity_column_exists adlist number
+ output='++ printf '\''.timeout 30000\nSELECT EXISTS(SELECT * FROM pragma_table_info('\''\'\'''\''%s'\''\'\'''\'') WHERE name='\''\'\'''\''%s'\''\'\'''\'');\n'\'' adlist number
++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp
1'
+ [[ ++ printf '.timeout 30000\nSELECT EXISTS(SELECT * FROM pragma_table_info('\''%s'\'') WHERE name='\''%s'\'');\n' adlist number
++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp
1 == \1 ]]
+ return 1
+ return
+ database_adlist_status 1 4
+ gravity_column_exists adlist status
+ output='++ printf '\''.timeout 30000\nSELECT EXISTS(SELECT * FROM pragma_table_info('\''\'\'''\''%s'\''\'\'''\'') WHERE name='\''\'\'''\''%s'\''\'\'''\'');\n'\'' adlist status
++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp
1'
+ [[ ++ printf '.timeout 30000\nSELECT EXISTS(SELECT * FROM pragma_table_info('\''%s'\'') WHERE name='\''%s'\'');\n' adlist status
++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp
1 == \1 ]]
+ return 1
+ return
+ echo ''

+ (( i++ ))
+ (( i < 1 ))
+ gravity_Blackbody=true
+ gravity_generateLocalList
+ echo '### Do not modify this file, it will be overwritten by pihole -g'
+ chmod 644 /etc/pihole/local.list
+ [[ -f /etc/openvpn/ipp.txt ]]
+ update_gravity_timestamp
+ output='++ printf '\''.timeout 30000\nINSERT OR REPLACE INTO info (property,value) values ('\''\'\'''\''updated'\''\'\'''\'',cast(strftime('\''\'\'''\''%%s'\''\'\'''\'', '\''\'\'''\''now'\''\'\'''\'') as int));'\''
++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp'
+ status=0
+ [[ 0 -ne 0 ]]
+ return 0
+ chown pihole:pihole /etc/pihole/gravity.db_temp
+ chmod g+w /etc/pihole /etc/pihole/gravity.db_temp
+ gravity_build_tree
+ local str
+ str='Building tree'
+ echo -ne '  [i] Building tree...'
  [i] Building tree...+ output='++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp '\''CREATE INDEX idx_gravity ON gravity (domain, adlist_id);'\'''
+ status=0
+ [[ 0 -ne 0 ]]
+ echo -e '\r  [✓] Building tree'
  [✓] Building tree
+ gravity_ShowCount
+ gravity_Table_Count gravity 'gravity domains' ''
+ local table=gravity
+ local 'str=gravity domains'
+ local num
++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp 'SELECT COUNT(*) FROM gravity;'
+ num=0
+ [[ gravity == \g\r\a\v\i\t\y ]]
+ local unique
++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp 'SELECT COUNT(*) FROM (SELECT DISTINCT domain FROM gravity);'
+ unique=0
+ echo -e '  [i] Number of gravity domains: 0 (0 unique domains)'
  [i] Number of gravity domains: 0 (0 unique domains)
+ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp 'INSERT OR REPLACE INTO info (property,value) VALUES ('\''gravity_count'\'',0);'
+ gravity_Table_Count vw_blacklist 'exact denied domains'
+ local table=vw_blacklist
+ local 'str=exact denied domains'
+ local num
++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp 'SELECT COUNT(*) FROM vw_blacklist;'
+ num=1
+ [[ vw_blacklist == \g\r\a\v\i\t\y ]]
+ echo -e '  [i] Number of exact denied domains: 1'
  [i] Number of exact denied domains: 1
+ gravity_Table_Count vw_regex_blacklist 'regex denied filters'
+ local table=vw_regex_blacklist
+ local 'str=regex denied filters'
+ local num
++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp 'SELECT COUNT(*) FROM vw_regex_blacklist;'
+ num=0
+ [[ vw_regex_blacklist == \g\r\a\v\i\t\y ]]
+ echo -e '  [i] Number of regex denied filters: 0'
  [i] Number of regex denied filters: 0
+ gravity_Table_Count vw_whitelist 'exact allowed domains'
+ local table=vw_whitelist
+ local 'str=exact allowed domains'
+ local num
++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp 'SELECT COUNT(*) FROM vw_whitelist;'
+ num=0
+ [[ vw_whitelist == \g\r\a\v\i\t\y ]]
+ echo -e '  [i] Number of exact allowed domains: 0'
  [i] Number of exact allowed domains: 0
+ gravity_Table_Count vw_regex_whitelist 'regex allowed filters'
+ local table=vw_regex_whitelist
+ local 'str=regex allowed filters'
+ local num
++ pihole-FTL sqlite3 /etc/pihole/gravity.db_temp 'SELECT COUNT(*) FROM vw_regex_whitelist;'
+ num=0
+ [[ vw_regex_whitelist == \g\r\a\v\i\t\y ]]
+ echo -e '  [i] Number of regex allowed filters: 0'
  [i] Number of regex allowed filters: 0
+ gravity_swap_databases
+ str='Swapping databases'
+ echo -ne '  [i] Swapping databases...'
  [i] Swapping databases...++ stat -f --format %a /etc/pihole
+ availableBlocks=249547609
++ stat --format %b /etc/pihole/gravity.db
+ gravityBlocks=208
+ oldAvail=false
+ '[' 249547609 -gt 416 ']'
+ '[' -f /etc/pihole/gravity.db ']'
+ oldAvail=true
+ mv /etc/pihole/gravity.db /etc/pihole/gravity_old.db
+ mv /etc/pihole/gravity.db_temp /etc/pihole/gravity.db
+ echo -e '\r  [✓] Swapping databases'
  [✓] Swapping databases
+ true
+ echo -e '  [✓] The old database remains available'
  [✓] The old database remains available
+ gravity_Cleanup
+ local error=
+ str='Cleaning up stray matter'
+ echo -ne '  [i] Cleaning up stray matter...'
  [i] Cleaning up stray matter...+ rm '/etc/pihole/pihole.*.txt'
+ rm '/etc/pihole/*.tmp'
+ rm /tmp/tmp.phgpb
+ rm '/tmp/*.ph-non-domains'
+ [[ true == true ]]
+ for file in "${piholeDir}"/*."${domainsExtension}"
+ [[ ! /etc/pihole/list.1.raw.githubusercontent.com.domains == *\/\e\t\c\/\p\i\h\o\l\e\/\*\.\d\o\m\a\i\n\s* ]]
+ rm -f '/etc/pihole/*.domains'
+ echo -e '\r  [✓] Cleaning up stray matter'
  [✓] Cleaning up stray matter
+ [[ -n '' ]]
+ echo ''

+ echo '  [✓] Done.'
  [✓] Done.

[PromoFaux]

@DL6ER can you rebase this one please? I have tried locally, but not 100% sure I did it right...

[DL6ER]

Done (albeit merge, not rebase)

[github-actions]

Conflicts have been resolved.

[PromoFaux]

[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
[✗] Status: raw.githubusercontent.com is blocked by one of your lists. Using DNS server 8.8.8.8 instead
[✓] Status: Retrieval successful
[✓] Parsed 178072 exact domains and 0 ABP-style domains (blocking, ignored 1 non-domain entries)
Sample of non-domain entries:
- 0.0.0.0
[i] List stayed unchanged