Thank you and dns.nextdns.io

[xfch7]

Thank you for the consolidated list you provide, this is very useful !

Server dns.nextdns.io is not blocked (although listed in https://github.com/curl/curl/wiki/DNS-over-HTTPS )

Is it deliberate for some reason ?

[jpgpi250]

my results:
nslookup dns.nextdns.io

Server:         127.0.0.1
Address:        127.0.0.1#53
Non-authoritative answer:
dns.nextdns.io  canonical name = steering.nextdns.io.
Name:   steering.nextdns.io
Address: 194.110.115.97
Name:   steering.nextdns.io
Address: 45.128.133.120

cat /var/www/html/DOHservers/DOHipv4.txt | grep 194.110.115.97
194.110.115.97

cat /var/www/html/DOHservers/DOHipv4.txt | grep 45.128.133.120
45.128.133.120

Both addresses are are available in the result file. DNS queries to obtain the IP address are retrieved, using a recursive unbound resolver, the resulting IP address is than verified, using a reverse lookup query on the OpenDNS server(s).

I live in Belgium, Is it possible you get different results (IP addresses) in your region?

[xfch7]

I live in Switzerland.
However, I use unbound in forward mode to a DoT server (SWITCH), maybe is it caused by that ?
All DNS or DoT requests are NATed to the firewall except for the VLAN from which I've done the following lookups

195.186.1.110 is a DNS server of one of the main ISP in Switzerland

> dns.nextdns.io
Server:		195.186.1.110
Address:	195.186.1.110#53

Non-authoritative answer:
dns.nextdns.io	canonical name = steering.nextdns.io.
Name:	steering.nextdns.io
Address: 178.255.153.47
Name:	steering.nextdns.io
Address: 159.100.248.193
Name:	steering.nextdns.io
Address: 2a00:11c0:5:613::3
Name:	steering.nextdns.io
Address: 2a04:c44:e00:8370:469:38ff:fe00:c9

With 8.8.8.8

> dns.nextdns.io
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
dns.nextdns.io	canonical name = steering.nextdns.io.
Name:	steering.nextdns.io
Address: 37.252.245.241
Name:	steering.nextdns.io
Address: 103.57.251.20
Name:	steering.nextdns.io
Address: 2a00:11c0:e:ffff:1::d
Name:	steering.nextdns.io
Address: 2a05:f480:1800:8ed:5400:2ff:fec8:7e46

So it seems this dns.nextdns.io has a lot of different IPs, however there might some basic principle I'm missing ? At the moment I just added this hostname in custom unbound options

server:
local-zone: "dns.nextdns.io" always_nxdomain

[jpgpi250]

When looking at my unbound logs, I can clearly see the resolvers that finally provide my answer (ns1.nextdns.io and ns2.nextdns.io)

step 1: find the IP for ns1.nextdns.io

nslookup ns1.nextdns.io
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   ns1.nextdns.io
Address: 45.90.28.1

step 2: lookup the addresses for dns.netxtdns.io, using the correct nameserver

nslookup dns.nextdns.io 45.90.28.1
Server:         45.90.28.1
Address:        45.90.28.1#53

Non-authoritative answer:
dns.nextdns.io  canonical name = steering.nextdns.io.
Name:   steering.nextdns.io
Address: 45.128.133.120
Name:   steering.nextdns.io
Address: 194.110.115.97

you can repeat the test, using the second nameserver (ns2.nextdns.io = 45.90.30.1), you'll get the same results.

My conclusion (assuming you get the same results fot the NS servers): Your provider (195.186.1.110) is changing the results into something they want you to use, as opposed to the real thing. This is a known practice some providers use to change or block access to some resources on the internet. A typical example would be blocking access to torrent sites (piratebay), mandatory in some countries. It appears your provider is using this method to let you believe you're actually using dns.nextdns.io, but you're really using something else, or there is an instance in between you and nextdns to manipulate some results.
My advise: implement a local recursive DNS resolver, such as (but not limited to) unbound. I've been using this for years, without any problems. Don't know what system you're using this on, here is a pretty simple configuration, used by many pihole users, with some additional information.

[xfch7]

This is quite interesting.

Using a recursive DNS server, or even asking directly 45.90.28.1 or 45.90.30.1, I don't get the same results.

I'm not very familiar with the concept, but might this be due to anycast DNS (and some round robin ?)

nslookup dns.nextdns.io 45.90.28.1
Server:		45.90.28.1
Address:	45.90.28.1#53

Non-authoritative answer:
dns.nextdns.io	canonical name = steering.nextdns.io.
Name:	steering.nextdns.io
Address: 178.255.153.47
Name:	steering.nextdns.io
Address: 159.100.248.193

nslookup dns.nextdns.io 45.90.30.1
Server:		45.90.30.1
Address:	45.90.30.1#53

Non-authoritative answer:
dns.nextdns.io	canonical name = steering.nextdns.io.
Name:	steering.nextdns.io
Address: 178.255.153.47
Name:	steering.nextdns.io
Address: 159.100.248.193

I then went to https://dnslookup.online/ and various IPs were reported, often very different.

In any case, using my own recursive DNS server is probably a good solution I should consider.

[xfch7]

Even more interesting : https://dnschecker.org/#A/dns.nextdns.io

Adding 45.90.30.1 as a custom DNS server returns different IPs, compared to both your or my results.

[jpgpi250]

Strange, different answers from the same nameservers in different regions...

There is really nothing I can do about this, since I always get the same result in my region, so it's probably a very good idea to add your unbound config

server:
local-zone: "dns.nextdns.io" always_nxdomain

Since I don't know the reason, I've opened a topic regarding this problem, you can read the question and possible answers here.

[xfch7]

Indeed, I also believe in this case the best solution is the nxdomain one.

Thank you very much for your time and, again for the list you are maintaining

[jrschat]

I see different IPs than both of you. Are you willing to share the scripts you have written to parse the lists used to create your four lists so that individual users can create "localized" lists?

[jpgpi250]

I'm currently in the process to consolidate and cleanup the scripts, as the code isn't very readable (read professional) in it's current state. For now you will have to download the lists as is, and add the IP addresses for dns.nextdns.io for your location.

The function I currently use to retrieve the IP addresses:

function getIP() {
# $1 domain
# $2 query type (A or AAAA)
[[ "$2" == "A" ]] && resolver=127.10.10.2 || resolver=fdaa:bbcc:ddee:2::5552
IFS=$'\n' read -r -d '' -a IParray < <( dig +tries=2 +time=5 @${resolver} -p 5552 +short $1 $2 && printf '\0' )
}

The resolvers I use, are local unbound listen addresses on port 5552, you need to change these into something that suits your needs.
I than use ipcalc and ipv6calc to validate the content of the array (result of the function) as valid IP address(es), and add them to the list.

Keeping your request in mind...

[jpgpi250]

In order to allow you to build your localized IP lists, I've added a sqlite3 database to the repository. The database contains the information, I use to execute the dig requests. A description of the content can be found in the updated manual.

Hope this helps a bit...

[jrschat]

Thank you very much. I was not expecting anything anytime soon. It will be this coming weekend before I can do anything with it on my end.

…

On Feb 28, 2021, at 04:56, jpgpi250 ***@***.***> wrote:  In order to allow you to build your localized IP lists, I've added a sqlite3 database to the repository. The database contains the information, I use to execute the dig requests. A description of the content can be found in the updated manual. Hope this helps a bit... — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[jrschat]

This is what I have come up with so far:

https://github.com/jrschat/PublicStuff/blob/master/DOH_database_script.sh

[jpgpi250]

Some of the DOH lists have been updated recently, as a result, the content of the database changes. I noticed (checked the database) that some entries are no longer in the lists, but still have an entry in the database (intentional, the timestamp field).

I would recommend to use the timestamp field to build the lists, as adding domains (and the resulting IPs) with an expired timestamp can (might OR might not) result in blocking IPs you don't want to block. Examples on how to use the timestamp field in sqlite3 queries are provided in the manual (section 5.2).

[jrschat]

I think I did right.

[jrschat]

I have a few more changes I will probably make before I call it done but I think I am getting close.
https://github.com/jrschat/PublicStuff/blob/master/DOH_database_script.sh

Thanks again for looking at it and for making the database.

[jpgpi250]

Looked at your script.
You don't need to add / resolve the entries in the cnameinfo table
Example:
Look at the first entry in the cnameinfo table: lux1.nixnet.xyz., domainlist_id: 101
now find the entry in the domainlist table: entry with unique_id 101 (is domainlist_id in the cnameinfo table): adblock.lux1.dns.nixnet.xyz

dig +short adblock.lux1.dns.nixnet.xyz ->
lux1.nixnet.xyz. (entry in the cnameinfo table)
104.244.78.231

dig +short lux1.nixnet.xyz. ->
104.244.78.231

The IPs are the same because adblock.lux1.dns.nixnet.xyz. is a CNAME for lux1.nixnet.xyz.

The entries in the cnameinfo table are only useful for users that which to create a DNS based list for use with sofware, such as pihole.

The end result of your script (including the domains in the cnameinfo table) is the same, but you're doing unnecessary dig requests.

[jrschat]

Interestingly, I get a different number of IPs (1 more if I include the CNAMES) in my list if I include the "duplicate" lookups.

[jpgpi250]

Only reason I can think of for this to happen is a query timeout for one of the digs.
Try NOT deleting DOHdig.txt and check if there is a query error
I'm using +tries=3 +time=4 (may need adjusting), unbound as a recursive resolver to get the IPs.

What is your resolver (@192.168.0.8)?

[jrschat]

It is my development PiHole instance that points upstream to my production pair that are both recursive unbound resolvers.

Changing it to one of the directly and increasing the tries to 3 brought the number up to the same. I haven't turned my IDS rules off to see if my full list of numbers will equal yours since I am blocking a couple TLDs.

[jpgpi250]

You shouldn't use pihole as resolver. If there is an entry in a blocklist (or a regex), you get 0.0.0.0 as a result, which you remove with your script (grep -E -v '^0..{0,11}$' DOHdup.txt > DOHip4.txt). You should use a dig command that directly queries unbound. IDS may impact the results, if there are active rules that block specific TLDs.

[jrschat]

Cool, that brought it closer, up to 271 from 268. My IDS was catching a few. Four more to go to match quantity.

Would you qualify any of these at query errors?
DOHdig.txt

[jpgpi250]

Doesn't look like there are errors in your result.

using todays db,
SELECT count(distinct domain) FROM "domainlist" WHERE timestamp is '1616822408';
gives 281 as a result. The IPv4 file in the repository only has 274 entries

earlier, I listed the function I use to get the IPs, the result is stored in an array, which you can than test to see if it's containing any elements. Dig +short doesn't always return a result -> empty result. This issue is all about regional differences, so it's very possible you will never get the same result (count).

Also, the db is always todays result of the script (script runs at 06 am CET), the IP lists are always one day old (yesterdays result), this gives me the option to make a correction, if something is faulty in the resulting lists.

[jrschat]

Cool, I think I am happy with what I have for my purposes then. Now just to figure out how to script the push of the list back into GitHub from my linux box.

[jpgpi250]

Latest entry is over 2 months old.
Users are adding unrelated issues to this issue, moved new issue and closed.