Dashboard header reports an incorrect status when running on docker

[scara]

Versions

Running docker tag image v5.2.1, $ docker exec -it pihole pihole -v:

• Pi-hole version is v5.2.1 (Latest: v5.2.1)
• AdminLTE version is v5.2.1 (Latest: v5.2.1)
• FTL version is v5.3.2 (Latest: v5.3.2)

Platform

Running using docker tag image v5.2.1,

• OS and version: Debian 10.5
• Platform: Docker on Raspberry Pi

Expected behavior

Status in the dashboard should show: "Active"

Actual behavior / bug

Status in the dashboard shows: "DNS service not running"

Steps to reproduce

Steps to reproduce the behavior:

1. Run the default image without adding any additional cap - no DHCP required
2. Point the browser to the web interface

Asking for the pihole status shows:

$ docker exec -it pihole pihole status
  [✗] DNS service is NOT listening

while if you add the following argument to the docker run command, --cap-add=SYS_PTRACE, you'll get:

$ docker exec -it pihole pihole status
  [✓] DNS service is listening
     [✓] UDP (IPv4)
     [✓] TCP (IPv4)
     [✓] UDP (IPv6)
     [✓] TCP (IPv6)

  [✓] Pi-hole blocking is enabled

This is a regression added w/ pi-hole/pi-hole@6009e86 since lsof requires to read /proc/*/stat. See ptrace(2).
When the cap is there:

$ docker exec -it pihole lsof -Pni:53
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
pihole-FT 835 pihole    4u  IPv4 121563      0t0  UDP *:53
pihole-FT 835 pihole    5u  IPv4 121564      0t0  TCP *:53 (LISTEN)
pihole-FT 835 pihole    6u  IPv6 121565      0t0  UDP *:53
pihole-FT 835 pihole    7u  IPv6 121566      0t0  TCP *:53 (LISTEN)

Shortly: lsof is not Docker friendly unless using CAP_SYS_PTRACE.
I'm not sure if the fix here is just to make a clear mention of the new cap requirement in the README or to find another way to provide that beautiful report.

HTH,
Matteo

[PromoFaux]

Thanks for the report. I'm not 100% this is reproducible - but I'm a bit green when it comes to docker.

But, short story - I don't have CAP_SYS_PTRACE added in my compose file, but the two commands run fine:

I'm currently running the :dev tag, but I don't think anything that would affect this has been added into that since we released the :v5.2.1 tag

edit: just switched back to :v5.2.1 and still unable to repro...

[scara]

Hi @PromoFaux,
thanks for the quick reply.

but I'm a bit green when it comes to docker

Me too 😉.
I'm using docker on a Pi4:

$ docker -v
Docker version 20.10.1, build 831ebea

and I'm running pihole not via a docker compose but a simple run:

$ docker run -d \
             --name pihole \
             --restart=unless-stopped \
             --hostname=pihole-container \
             --dns=127.0.0.1 --dns=8.8.8.8 \
             --add-host="pi401 pi401.scaramuccia.home:127.0.0.1" \
             --cap-add=NET_BIND_SERVICE \
             --cap-add=SYS_NICE \
             --cap-add=SYS_PTRACE \
             --cap-drop=NET_RAW \
             -p 53:53/udp -p 53:53/tcp \
             -p 80:80/tcp \
             -p 443:443/tcp \
             -v /srv/pihole/etc/pihole:/etc/pihole \
             -v /srv/pihole/etc/dnsmasq.d:/etc/dnsmasq.d \
             -e ServerIP=192.168.0.5 \
             -e IPv6=False \
             -e DNS1=8.8.8.8 \
             -e DNS2=8.8.4.4 \
             -e DNSMASQ_USER=pihole \
             -e TZ=Europe/Rome \
             -e WEBPASSWORD=******** \
             -e VIRTUAL_HOST=pi401.scaramuccia.home \
             pihole/pihole:v5.2.1

If I remove the --cap-add=SYS_PTRACE \ above, I get the issue - the cap has been added just an hour ago before opening this issue.

HTH,
Matteo

[PromoFaux]

Hmm, I'm on a Synology Nas with docker at v18.09.8, not sure how much of a difference that makes.
compose file as follows (n.b, i'm using macvlan for the network layer, so don't need to do anything with port variables

version: '3.5'
services:

  pihole:
    container_name: pihole
    hostname: pihole-docker
    depends_on:
      - unbound
    image: pihole/pihole:dev
    environment:
      - TZ=Europe/London
      - ServerIP=192.168.1.253
      #- DNS1=8.8.8.8 #not needed in dev image
      #- DNS2=8.8.4.4 #not needed in dev image
      - PIHOLE_DNS_=8.8.8.8;8.8.4.4;127.0.0.1#5353 #new in dev image
      - REV_SERVER=true
      - REV_SERVER_DOMAIN=lan
      - REV_SERVER_TARGET=192.168.0.1
      - REV_SERVER_CIDR=192.168.0.0/16
      - SKIPGRAVITYONBOOT=1
    volumes:
      - ./pihole/configs/pihole/:/etc/pihole/
      - ./pihole/configs/dnsmasq.d:/etc/dnsmasq.d/
    mac_address: d0:ca:ab:cd:ef:fe
    networks:
      home:
        ipv4_address: 192.168.1.253
    restart: always

Paging @diginc for thoughts

[hamtaai]

I also ran into this after upgrading from v5.1.2 to v5.2.1 (Raspberry Pi 3 with raspbian). Adding CAP_SYS_PTRACE fixed it for me.

[scara]

Adding a cross-reference about the lsof requirement: pi-hole/pi-hole#3870.

HTH,
Matteo

[aman-sagoo]

I am also seeing this issue , only happens when DNSMASQ_USER=pihole , if left root this issue is not present.

[pralor-bot]

This issue has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/dns-service-not-running/42450/3

[benlenau]

This env says "experimental" - And eventhough the webui states "DNS service not running" it's running and 53 requests are handled like they always have. I like seeing more and more containers focusing on permissions and --cap-add=SYS_PTRACE kindda defeats the purpose of this env.

[scara]

Hi @benlenau,
DNSMASQ_USER has not a direct relation with this issue: I mean, pi-hole/pi-hole@6009e86#diff-7220ce3ef224faa8ffe3caf9ae1c7cb19db28cbd896fda113412490f308f658bR290 introduced lsof to improve the status reporting.
Being marked as experimental maybe that ENV VAR is not widely used and a few people fall into this issue.

BTW I agree w/ you, something should be done:
a. find another nice way to do the job of the new status report. At least use the previous method when not enough caps are found to run lsof
b. add CAP_SYS_PTRACE
c. drop the ENV VAR

My vote is:
a. +1
b. +0.1 as a temporary workaround, +0.5 when the host is running few not publicly exposed containers
c. -1

HTH,
Matteo

[benlenau]

Hi @benlenau,

DNSMASQ_USER has not a direct relation with this issue: I mean, pi-hole/pi-hole@6009e86#diff-7220ce3ef224faa8ffe3caf9ae1c7cb19db28cbd896fda113412490f308f658bR290 introduced lsof to improve the status reporting.

Being marked as experimental maybe that ENV VAR is not widely used and a few people fall into this issue.

BTW I agree w/ you, something should be done:

a. find another nice way to do the job of the new status report. At least use the previous method when not enough caps are found to run lsof

b. add CAP_SYS_PTRACE

c. drop the ENV VAR

My vote is:

a. +1

b. +0.1 as a temporary workaround, +0.5 when the host is running few not publicly exposed containers

c. -1

HTH,

Matteo

My bad. Also hoping for a. :-)

[scoiatael]

FYI something related and very weird happens when running PiHole on K3s (kubernetes distribution) - lsof takes ages to finish. This in turn causes very slow startup and slow loading of admin interface.

Maybe we could move to netstat -tlp | grep :53 as a replacement for lsof?

[AlBundy33]

I've installed today rocky linux 8.5 and moved my pihole installation from my rapspberry pi to amdoxker container on my nuc.
Inhave the same "problem" - pihole works fine, only the status in the web ui is wrong.

setting cap or user to root seems to workaround the issue but it would be nice to have a better solution.

[PromoFaux]

I've added a note to the readme for the time being, as we recently defaulted to running pihole-FTL as pihole, rather than root, and this was not caught during my (brief!) testing.

[DL6ER]

We should indeed replace lsof as suggested by @scoiatael. However, I prefer ss as netstat is considered obsolete.

Check out my branch tweak/ss.
If it doesn't work, try tweak/netstat as alternative.

Note 1: These are poof-of-concept branches. I did not check dependencies that may need to be installed. netstat will need net-tools, ss is included in iproute2 which is, e.g., a dependency of ubuntu-minimal but this may not be a case for all operating systems. If you run into issues when testing, please install the dependencies and leave a comment here!

Note 2: I prefer the long-options as it makes it easier to see what is happening without having to go the the man page. --numeric ensures we are not trying to resolve hostnames and send out a couple of PTR queries every time this runs.

Note 3: Once we have settled to one of the solutions, I will replace the remaining use of lsof also in the debug script and remove the dependency.

[pralor-bot]

This issue has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/dns-service-not-running-on-raspberry-pi-4b/52195/14

[scara]

Hi @DL6ER,

Check out my branch tweak/ss.
If it doesn't work, try tweak/netstat as alternative.

where did you push your PoCs?

Edit: found them ;):

• https://github.com/pi-hole/pi-hole/tree/tweak/ss
• https://github.com/pi-hole/pi-hole/tree/tweak/netstat

TIA,
Matteo

[PromoFaux]

If you want to try these in docker easily - you can use the :nightly tag, and run pihole checkout core tweak/ss or pihole checkout core tweak/netstat (pihole checkout is disabled in other builds)

[PromoFaux]

tweak/ss - works without adding any additional deps to the container
tweak/netstat - requires net-tools to be added to the container.

Obviously tweak/ss is the one to go for if netstat is considered obsolete (I'll remove the note about adding the additional CAP once we merge it into dev)

[PromoFaux]

This will be fixed shortly in the next release

[jimaldon]

Looks like it was fixed in pihole v5.8
Any idea when the latest tagged image in https://hub.docker.com/r/pihole/pihole/ would include v5.8?

[PromoFaux]

At the moment the best answer I can give you is "when it's ready", was hoping to tag straight away but have come up against a snag

[PromoFaux]

Should now be fixed... please report back!

[amigthea]

I can confirm the fix, thank you for your work and have a great day!

[jimaldon]

Fixed for me, thanks!

[manfromdownunder]

Confirm that the issue resolved in docker tag 2022.01.1. Thankyou for the prompt fix!

[AlBundy33]

same here 👍

[Bruce17]

Can also confirm docker tag 2022.01.1 is running fine on top of Kubernetes cluster.

[scara]

Hi @PromoFaux and @DL6ER,

Should now be fixed... please report back!

2022.01.1 works fine on my side too i.e. w/o the need of --cap-add=SYS_PTRACE: I think the issue should be closed.

Thanks for the hard work!
Matteo

[PromoFaux]

Great stuff, thanks for letting us know!