DNS queries from other docker containers fail when server IP is used in resolv.conf

[mfncl99]

This is an issue with the latest debian dev tag. May also be present in other tags, haven't tested these.

DNS queries initiated by any docker container (including pihole) on the host fail when the hosts /etc/resolv.conf is set to the local server IP (in my case, 10.50.10.3), despite pi-hole logs showing a successful lookup event.

$ docker exec mycontainer ping google.com
ping: bad address 'google.com'

Feb 12 11:59:50 dnsmasq[539]: 91 172.17.0.1/55868 query[A] google.com from 172.17.0.1
Feb 12 11:59:50 dnsmasq[539]: 91 172.17.0.1/55868 cached google.com is 172.217.10.110
Feb 12 11:59:50 dnsmasq[539]: 92 172.17.0.1/55868 query[AAAA] google.com from 172.17.0.1
Feb 12 11:59:50 dnsmasq[539]: 92 172.17.0.1/55868 forwarded google.com to 187.37.39.39
Feb 12 11:59:50 dnsmasq[539]: 92 172.17.0.1/55868 forwarded google.com to 185.37.37.37
Feb 12 11:59:50 dnsmasq[539]: 92 172.17.0.1/55868 reply google.com is 2607:f8b0:4006:811::200e

I verified another container can see port 53 on the host IP:

$ docker exec mycontainer nc -zv 10.50.10.3 53
10.50.10.3 (10.50.10.3:53) open

All DNS queries from any other client outside of Docker are successful - pihole is correctly configured, no errors in pihole.log.

I was able to fix this by updating the hosts /etc/resolv.conf to point to the docker interface (instead of the IP of the host, 10.50.10.3):

/etc/resolv.conf
nameserver 172.17.0.1

Setting Google DNS on the hosts /etc/resolv.conf also appears to be a valid workaround.

Below is my Docker container config. Since the DNS server picks up and responds to the DNS query, it seems there's something odd going on with routing that prevents containers from receiving DNS responses back from pihole.

Unsure on if this would be classified as a bug with this container or is expected behavior due to Dockers routing design.

If expected behavior, it may be useful to document this as a note for other users within README.md

docker create \
    --name="pihole" \
    --restart=always \
    -e ServerIP=10.50.10.3 \
    -e PROXY_LOCATION=pihole \
    -e VIRTUAL_HOST=pihole.home \
    -e VIRTUAL_PORT=80 \
    -e WEBPASSWORD=XXX \
    -e DNS1=185.37.37.37 \
    -e DNS2=187.37.39.39 \
    --add-host "pihole.home:10.50.10.3" \
    -v /home/mark/config/pihole/:/etc/pihole/ \
    -v /home/mark/config/dnsmasq.d/:/etc/dnsmasq.d/ \
    -v /home/mark/config/dnsmasq.conf:/etc/dnsmasq.conf \
    -e TZ=America/New_York \
    -p 53:53/tcp \
    -p 53:53/udp \
    -p 8053:80/tcp \
    diginc/pi-hole:debian_dev

[diginc]

Has this been resolved by v3.3?

[solazs]

Nope.

I'm trying to run smokeping next to pi-hole and got the same error w/ pihole 3.3

[edgd1er]

I have the same events with the armhf. The workarounds ( pihole docker private ip 172.... Or external dns server (ie my router dns)) .

I tried a "docker network connect pihole_default mycontainer". I could ping on container's name but not on external names.

Embedded dns may be a point to analyze. I did not find the latest version of that point (18.03)
https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/

Note: If you need access to a host’s localhost resolver, you must modify your DNS service on the host to listen on a non-localhost address that is reachable from within the container.

Note: The DNS server is always at 127.0.0.11.

I noticed that problem after an unwanted reboot last tuesday (07/17/18) :( the container was created beginning of june. Raspberry Updates are installed as they re out.

The probleme seems to be because we are running containers on the same host that run pihole in a container.

[diginc]

I recently had to do this for work. I faked out some DNS records for other containers using a docker pi-hole container containing the desired dns records.

The trick was just editing the docker host service (https://docs.docker.com/config/daemon/#configure-the-docker-daemon) to use localhost as DNS (I also added a backup dns because chicken before egg). the first--dns flag was 127.0.0.1 and the other was a standard upstream dns server. I made sure the pi-hole container was set to auto-start and was created before adding this flag to the docker daemon so that it'd be there nearly instantly when restarting the docker daemon service.

There's no worry about what networks are connected to which with this approach or configuring individual containers' DNS. It's not exactly pretty but it worked short term for the testing I was doing.

[edgd1er]

edit: changed daemon dns from 127.0.0.1 to pihole docker network's ip 172.20.0.2 + need to join pihole's network for each container that need name resolution as I'm using docker-compose to create containers.

docker network join pihole_default mycontainername

I wanted to be able to resolve public servers names and local names stored in a file hosts.domain in containers running on the same host as pihole. a custom file for dnsmasq is already defined containing:

addn-hosts=/etc/pihole/hosts.mission

i got a result with this 3 steps fix:

• on the host, add two entries in /etc/docker/daemon.json =>

     "dns":["published.dns.ip","upstream.server.ip"],
      "dns-search": ["mydomain.tld"]

• in the pihole docker-compose yml:

    dns:
    - 127.0.0.1

• in all other docker-compose.yml of the host running pihole: the pihole host ip and the domain

     dns:
      - "172.20.0.2"
     dns_search: "mydomain.tld"

[melroy89]

I'm basically using the docker compose config example file.

To this (webservice runs behind Nginx):

version: "3"
services:
  pihole:
    image: pihole/pihole:latest
    dns:
      - 127.0.0.1
      - 1.0.0.1
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "67:67/udp"
      - "8053:80/tcp"
      - "8054:443/tcp"
    environment:
      ServerIP: 192.168.1.20
      TZ: Europe/Amsterdam
      DNS1: 1.0.0.1
      DNS2: 1.1.1.1
      WEBPASSWORD: secure_password
      # jwilder/proxy envs, see readme for more info
      PROXY_LOCATION: pihole
      VIRTUAL_HOST: proxy.melroy.org
      VIRTUAL_PORT: 80
    extra_hosts:
      - 'proxy.melroy.org:192.168.1.20'
    volumes:
      - '/etc/pihole/:/etc/pihole/'
      - '/etc/dnsmasq.d:/etc/dnsmasq.d'
      - '/var/log/pihole.log:/var/log/pihole.log'
    restart: always

I created a daemon.json file, /etc/docker/daemon.json:

{
   "dns": ["192.168.1.20", "1.0.0.1"]
}

Pi-hole is running fine. I restart docker via: sudo systemctl restart docker.
Finally I run:

$ docker run busybox cat /etc/resolv.conf
nameserver 192.168.1.20
nameserver 1.0.0.1

Which looks fine by me. However, when I try to do a nslookup for example, it will not use 192.168.1.20 😢

$ docker run busybox nslookup google.com
Server:		1.0.0.1
Address:	1.0.0.1:53

Non-authoritative answer:
Name:	google.com
Address: 172.217.20.110

*** Can't find google.com: No answer

As you can see it will try to actually use the secondary DNS server (1.0.0.1) instead as fail-cover.
I can ping the server from out the container:

$ docker run busybox ping 192.168.1.20
PING 192.168.1.20 (192.168.1.20): 56 data bytes
64 bytes from 192.168.1.20: seq=0 ttl=64 time=0.139 ms
64 bytes from 192.168.1.20: seq=1 ttl=64 time=0.154 ms
64 bytes from 192.168.1.20: seq=2 ttl=64 time=0.151 ms
64 bytes from 192.168.1.20: seq=3 ttl=64 time=0.156 ms
^C
--- 192.168.1.20 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.139/0.150/0.156 ms

Some extra info.

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
c367a2632062        bridge              bridge              local
4fd7c4c7cd28        host                host                local
7394d99ac8bd        melroy_default      bridge              local
16a5bf4e5a94        none                null                local

...
docker --version
Docker version 18.09.1, build 4c52b90

...
Pi-hole Version v4.1.1 (within Docker container)

$ docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                   PORTS                                                                                                     NAMES
ffb5d072a637        pihole/pihole:latest   "/s6-init"               5 hours ago         Up 7 minutes (healthy)   0.0.0.0:53->53/udp, 0.0.0.0:53->53/tcp, 0.0.0.0:67->67/udp, 0.0.0.0:8053->80/tcp, 0.0.0.0:8054->443/tcp   melroy_pihole_1


Running Debian 9 (Stretch) - systemd

I want to that all my Docker images will use my Pi-Hole DNS. How? What do I wrong?

[diginc]

Docker networking is a little weird and is blocking hairpin port forwards by default. edgd1er covered how to do this accurately as far as I can tell after experimenting a little, look over his instructions and try those.

I would recommend adding a static IP to the pi-hole container and making sure you join all containers that need to use Pi-hole's DNS to the network your docker-compose created : melroy_default.

[melroy89]

@diginc @edgd1er

docker network join pihole_default mycontainername

TLDR; I need a working solution without manual steps in between.

This is not acceptable for every container that need Pi-Hole DNS. Since the Docker containers that need this are created by another program (called Gitlab Runner). Which is out my control, Gitlab Runner has control of the creation as well as maintenance of those containers.

So, currently by Gitlab continuous integration can't access it's own host, which is due to the missing bridge-mode support on my router/modem of my ISP (https://community.t-mobile.nl/t-mobile-thuis-internet-492/hg659-bridge-mode-298267).

Long story short, I can NOT use global DNS servers (like 1.1.1.1 or 8.8.8.8) for my local network services (I run a server with some services like Nginx, Gitlab, Nextcloud, etc and of course Pi-Hole). If you port forward a port in your router, you can access it out of your LAN and use a domain name (DNS to my external IP address), normally you can also use it internally using the same domain name.

However, with my T-mobile modem/router, I can't. Therefor I need to use my own DNS server, which resolves my domain-name to my local IP address of the server in order to get it working. Without using my DNS it will resolve to my external IP address, and this can not be handled by my router/modem.

Meaning, a continuous integration Docker image from my own Gitlab server, can't find it's own Gitlab server (it will not able to resolve it), and therefor can not git clone it's own repository, before you start your test cases (or whatever you like to do within your CI).

[diginc]

TLDR; I need a working solution without manual steps in between.

Add it to your docker-compose list of external networks.

[ramblingenzyme]

Is there a proper fix for this yet?

[edgd1er]

hi @ramblingenzyme ,

I recently set up a second pihole on a debian system. For an unknown reason, the ipv4_address setting in the pihole docker-compose was inufficient.
Setting the pihole container address in daemon.json was not a solution. But if I add the docker gateway, it worked very well. IPAM subnet is 172.20.0.0/16, IPAM gateway is 172.20.0.1

"dns":["172.20.0.1","other.pihole.ip","my.modem.ip"],

One change, and all dns resolutions in containers are more quicker.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Please comment or update this issue or it will be closed in 5 days.