Upstreams with Internal Hostname Supplied in /etc/hosts Stopped Working on v0.107.31

[siaush]

Prerequisites

• I have checked the Wiki and Discussions and found no answer

• I have searched other issues and found no duplicates

• I want to report a bug and not ask a question

Operating system type

Linux, Other (please mention the version in the description)

CPU architecture

AMD64

Installation

Docker

Setup

On one machine

AdGuard Home version

v0.107.31 and edge

Description

What did you do?

• Upgraded AGH from v0.107.29 to v0.107.31, and internal upstream DNS servers specified as internal hostnames in the AGH Upstream DNS servers list stopped working.

• I have 2 AGH nodes running on 2 separate Docker hosts, and their settings are synchronised with Adguardhome-Sync.

• I also have 2 internal DNS servers nodes, running on 2 Docker hosts, one each on the same Docker host as each of the AGH containers.

• My Upstream DNS servers are specified as such in both of the AGH nodes:

  udp://private-upstream-1:5335
  udp://private-upstream-2:5335
  

• The actual IP addresses of private-upstream-1 and private-upstream-2 are specified in the docker-compose.yml file for each of the AGH nodes, using the extra_hosts directive.

• The reason I did not specify the upstream IP addresses directly in the AGH Upstream DNS is, both of the upstream DNS have different IP address when accessed from different AGH nodes that are hosted on different Docker host.

• An example of the extra_hosts directive in the docker-compose.yml file are:
  AGH Node 1:

  extra_hosts:
        - "private-upstream-1:10.0.0.1" 
        - "private-upstream-2:192.168.0.2" 
  

  AGH Node 2:

  extra_hosts:
        - "private-upstream-1:192.168.1.3" 
        - "private-upstream-2:10.0.1.4" 
  

Expected result

• Internal upstream DNS specified with internal hostnames should still be reachable from AGH.

• This works up until v0.107.29, and stops working on v0.107.31 as well as the edge version.

Actual result

• Both of the Internal upstream DNS specified as internal hostnames in the Upstream DNS servers throwing error: Upstream "udp://private-upstream-N:5335" does not respond to test requests and may not work properly.
• Tried with TCP connections and it does not work either.

Additional information

• Both private-upstream-1 and private-upstream-2 are reachable by ping and nslookup resolves successfully to the IP addresses specified in docker-compose.yml from inside the AGH containers.

• Replacing private-upstream-1 and private-upstream-2 with their actual IP addresses in the Upstream DNS Servers configuration works as well, but not feasible as the settings are synchronised across nodes, and not all IP addresses are accessible from both nodes.

• Rolling back the containers to v0.107.29 and the issue above goes away.

Thanks!

[ainar-g]

Hello and thanks for the thorough report. This is most likely because in v0.107.30 we've fixed a long-standing issue where the bootstrap IP addresses weren't used to resolve the hosts in udp:// upstreams. I'm assuming that docker-compose exposes its own DNS server on an address, and you should add that address to the Bootstrap DNS servers field so that it can resolve those hostnames.

[siaush]

Thanks for the quick reply.
By any chance do you know what is the address where docker-compose exposes the DNS server on? I have tried the Docker host's IP from within the container, but it is unreachable.

The extra_hosts directive adds the hostname and IP address combo into the container's /etc/hosts file, so I am kinda sceptical if the information will be available to the Docker host.

I did try resolving the private DNS server hostnames name using the AGH container's internal DNS (on 127.0.0.11), but too bad it is not resolvable as well.

By any chance AGH will allow resolving of upstream DNS server hostnames using the bootstrap DNS, plus the entries in the /etc/hosts file on the container? That would resolve the problem.

Thanks.

[webocalypse]

Hi,

it seems like you are having a similar problem to what i had.
I explained how i fixed it in this comment: #5874 (comment)

I hope this helps.

[siaush]

Hi @webocalypse, thanks for your reply.
Unfortunately adding 127.0.0.11:53 to the bootstrap DNS only partially resolves the issue for me.

This is because only 1 of the 2 upstream DNS nodes are created in the same Docker Compose stack, where the other node is on a separate Docker host altogether. That was the reason that I have added additional hosts entry via the extra_hosts directive.

So at any one time, AGH is only able to resolve the private upstream DNS hostname if the particular node is created in the same Docker Compose stack.

As of now, I am thinking the best bet to resolve this is to allow resolution of upstream DNS hostnames from both the /etc/hosts file and the bootstrap DNS.

Thanks :)

[ainar-g]

It does seem logical that the /etc/hosts data should be used for resolving these hostnames, and it seems like we've broken that feature. @EugeneOne1, I feel like we could use upstream.Options.ServerIPAddrs for this?

[ainar-g]

@siaush, this should be fixed as of v0.107.33. Can you please check?

[siaush]

@siaush, this should be fixed as of v0.107.33. Can you please check?

Thanks for the update @ainar-g, I am now running v0.107.33 on both of my Docker hosts.

They needed a little modification to work as they were in v0.107.29 - which is to add the AGH container's internal DNS server (127.0.0.11) to the bootstrap DNS list. This is probably because currently I do not include the IP address of the upstream DNS container if they are on the same Compose stack as AGH, in the extra_hosts directive. Only the upstream DNS that are located on the different Compose stack and host are included.

Other than that, it is working as it was in v0.107.29 now.

Thank you very much :)