On startup, image tries to CHOWN/CHGRP bind mounted directories and fails causing DNS resolution and admin interface to be unresponsive

[muzzah]

Versions

pihole | 2021-01-03T12:40:06.752789247Z Pi-hole version is v5.2.2 (Latest: v5.2.2)
pihole | 2021-01-03T12:40:06.831068126Z AdminLTE version is v5.2.2 (Latest: v5.2.2)
pihole | 2021-01-03T12:40:06.886936281Z FTL version is v5.3.4 (Latest: v5.3.4)

Platform

• OS and version: Ubuntu 20

• Platform: VPS - Exoscale

Expected behavior

Pihole starts up on a fresh install and has started to serve DNS requests while making admin interface accessible

Actual behavior / bug

Cannot access admin interface, DNS resolution fails and VPS host becomes unusable

Steps to reproduce

Steps to reproduce the behavior:

Follow the guide for installing behind nginx proxy.

Debug Token

I cannot do this because I cannot use DNS after install

Some of the error logs I see

pihole         | 2021-01-03T12:30:53.091653579Z chown: cannot access '': No such file or directory
pihole         | 2021-01-03T12:30:53.185449793Z chmod: cannot access '': No such file or directory
pihole         | 2021-01-03T12:30:53.209101507Z chown: changing ownership of '/etc/pihole': Input/output error
pihole         | 2021-01-03T12:30:53.279831325Z chown: cannot access '/etc/pihole/dhcp.leases': No such file or directory

Full log of startup

pihole         | 2021-01-03T12:56:30.220300775Z Stopping lighttpd
pihole         | 2021-01-03T12:56:30.227229652Z Stopping pihole-FTL
pihole         | 2021-01-03T12:56:30.230797341Z Stopping cron
pihole         | 2021-01-03T12:56:30.237511026Z [cont-finish.d] executing container finish scripts...
pihole         | 2021-01-03T12:56:30.239210192Z [cont-finish.d] done.
pihole         | 2021-01-03T12:56:30.239858598Z [s6-finish] waiting for services.
pihole         | 2021-01-03T12:56:30.455550480Z [s6-finish] sending all processes the TERM signal.
pihole         | 2021-01-03T12:56:33.463010373Z [s6-finish] sending all processes the KILL signal and exiting.
pihole         | 2021-01-03T12:56:45.684123929Z [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
pihole         | 2021-01-03T12:56:45.794940464Z [s6-init] ensuring user provided files have correct perms...exited 0.
pihole         | 2021-01-03T12:56:45.795835350Z [fix-attrs.d] applying ownership & permissions fixes...
pihole         | 2021-01-03T12:56:45.801612089Z [fix-attrs.d] 01-resolver-resolv: applying... 
pihole         | 2021-01-03T12:56:45.802001231Z [fix-attrs.d] 01-resolver-resolv: exited 0.
pihole         | 2021-01-03T12:56:45.802700762Z [fix-attrs.d] done.
pihole         | 2021-01-03T12:56:45.803986089Z [cont-init.d] executing container initialization scripts...
pihole         | 2021-01-03T12:56:45.804667817Z [cont-init.d] 20-start.sh: executing... 
pihole         | 2021-01-03T12:56:45.906309629Z  ::: Starting docker specific checks & setup for docker pihole/pihole
  [✓] Update local cache of available packages
pihole         | 2021-01-03T12:56:49.239905171Z   [i] Existing PHP installation detected : PHP version 7.3.19-1~deb10u1
pihole         | 2021-01-03T12:56:50.149769927Z 
pihole         | 2021-01-03T12:56:50.149791337Z   [i] Installing configs from /etc/.pihole...
pihole         | 2021-01-03T12:56:50.153658708Z   [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [✓] Copying 01-pihole.conf to /etc/dnsmasq.d/01-pihole.conf
pihole         | 2021-01-03T12:56:53.935379208Z chown: cannot access '': No such file or directory
pihole         | 2021-01-03T12:56:53.968982549Z chmod: cannot access '': No such file or directory
pihole         | 2021-01-03T12:56:53.997587161Z chown: changing ownership of '/etc/pihole': Input/output error
pihole         | 2021-01-03T12:56:54.073833649Z chown: cannot access '/etc/pihole/dhcp.leases': No such file or directory
pihole         | 2021-01-03T12:56:59.543410225Z Existing DNS servers detected in setupVars.conf. Leaving them alone
pihole         | 2021-01-03T12:56:59.567439679Z ::: Pre existing WEBPASSWORD found
pihole         | 2021-01-03T12:56:59.574866557Z DNSMasq binding to default interface: eth0
pihole         | 2021-01-03T12:57:06.919695018Z Added ENV to php:
pihole         | 2021-01-03T12:57:06.922134802Z 			"PHP_ERROR_LOG" => "/var/log/lighttpd/error.log",
pihole         | 2021-01-03T12:57:06.922177503Z 			"ServerIP" => "[Elastic IP]",
pihole         | 2021-01-03T12:57:06.922184549Z 			"VIRTUAL_HOST" => "pihole.mustafa.lol",
pihole         | 2021-01-03T12:57:09.705166620Z Using IPv4 and IPv6
pihole         | 2021-01-03T12:57:09.718127790Z ::: Preexisting ad list /etc/pihole/adlists.list detected ((exiting setup_blocklists early))
pihole         | 2021-01-03T12:57:09.779110978Z https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
pihole         | 2021-01-03T12:57:09.779154533Z https://mirror1.malwaredomains.com/files/justdomains
pihole         | 2021-01-03T12:57:10.311942399Z ::: Testing pihole-FTL DNS: FTL started!
pihole         | 2021-01-03T12:57:10.333964629Z ::: Testing lighttpd config: Syntax OK
pihole         | 2021-01-03T12:57:10.334682558Z ::: All config checks passed, cleared for startup ...
pihole         | 2021-01-03T12:57:10.335161122Z  ::: Docker start setup complete
pihole         | 2021-01-03T12:57:12.291756752Z   [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range
pihole         | 2021-01-03T12:57:12.915302637Z 
  [✓] Preparing new gravity database
pihole         | 2021-01-03T12:57:13.927821217Z   [i] Using libz compression
pihole         | 2021-01-03T12:57:13.927844483Z 
pihole         | 2021-01-03T12:57:13.928255759Z   [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
pihole         | 2021-01-03T12:57:14.665581528Z   [i] Received 58278 domains
pihole         | 2021-01-03T12:57:15.677900015Z 
pihole         | 2021-01-03T12:57:15.677933981Z   [i] Target: https://mirror1.malwaredomains.com/files/justdomains
  [✗] Status: Not found
pihole         | 2021-01-03T12:57:16.205240859Z   [✗] List download failed: no cached list available
pihole         | 2021-01-03T12:57:16.205282933Z 
  [✓] Storing downloaded domains in new gravity database
  [✓] Building tree
  [✓] Swapping databases
pihole         | 2021-01-03T12:57:22.887617011Z chmod: changing permissions of '/etc/pihole': Input/output error
pihole         | 2021-01-03T12:57:23.703969843Z   [i] Number of gravity domains: 58278 (58278 unique domains)
pihole         | 2021-01-03T12:57:25.068509268Z   [i] Number of exact blacklisted domains: 0
pihole         | 2021-01-03T12:57:25.434874218Z   [i] Number of regex blacklist filters: 0
pihole         | 2021-01-03T12:57:25.751512942Z   [i] Number of exact whitelisted domains: 0
pihole         | 2021-01-03T12:57:26.030436803Z   [i] Number of regex whitelist filters: 0
  [✓] Cleaning up stray matter
pihole         | 2021-01-03T12:57:26.683052814Z 
pihole         | 2021-01-03T12:57:27.188123214Z   [✓] DNS service is listening
pihole         | 2021-01-03T12:57:27.191439461Z      [✓] UDP (IPv4)
pihole         | 2021-01-03T12:57:27.194597964Z      [✓] TCP (IPv4)
pihole         | 2021-01-03T12:57:27.198304111Z      [✓] UDP (IPv6)
pihole         | 2021-01-03T12:57:27.200611353Z      [✓] TCP (IPv6)
pihole         | 2021-01-03T12:57:27.200627961Z 
pihole         | 2021-01-03T12:57:27.248101579Z   [✓] Pi-hole blocking is enabled
pihole         | 2021-01-03T12:57:27.321900060Z   Pi-hole version is v5.2.2 (Latest: v5.2.2)
pihole         | 2021-01-03T12:57:27.397419087Z   AdminLTE version is v5.2.2 (Latest: v5.2.2)
pihole         | 2021-01-03T12:57:27.433083838Z   FTL version is v5.3.4 (Latest: v5.3.4)
pihole         | 2021-01-03T12:57:27.435489043Z [cont-init.d] 20-start.sh: exited 0.
pihole         | 2021-01-03T12:57:27.437017579Z [cont-init.d] done.
pihole         | 2021-01-03T12:57:27.439345261Z [services.d] starting services
pihole         | 2021-01-03T12:57:27.451920593Z Starting crond
pihole         | 2021-01-03T12:57:27.456670333Z Starting pihole-FTL (no-daemon) as root
pihole         | 2021-01-03T12:57:27.458267168Z Starting lighttpd
pihole         | 2021-01-03T12:57:27.461777517Z [services.d] done.

Additional context

I currently have a setup where multiple containers live behind a nginx proxy (as per the example on this site and docker file below). I am also using a S3 (comptiable service) mounted bucket for storing the Pihole files necessary for function (bind mounted through docker-compose). I also have a public registered domain for pihole that points to an elastic IP which in turn points to my VPS

Elastic IP -> VPS [Has a different public IP from elastic IP pointing to it but the Elastic IP is used to access pihole and other services]

The folders which are bind mounted are set root:root ownership. In the startup logs, I see operations by pihole which fail
as per above.

This is my docker-compose file

version: '3' 

services:

  proxy:
    image: jwilder/nginx-proxy:alpine
    labels:
      - "com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy=true"
    container_name: nginx-proxy
    ports:
      - 80:80
      - 443:443
    volumes:
      - /mnt/proxy-files/conf.d:/etc/nginx/conf.d:rw
      - /mnt/proxy-files/vhost.d:/etc/nginx/vhost.d:rw
      - /mnt/proxy-files/html:/usr/share/nginx/html:rw
      - /mnt/proxy-files/certs:/etc/nginx/certs:ro
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/tmp/docker.sock:ro
    restart: unless-stopped

  letsencrypt:
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: letsencrypt
    depends_on:
      - proxy
    volumes:
      - /mnt/proxy-files/certs:/etc/nginx/certs:rw
      - /mnt/proxy-files/vhost.d:/etc/nginx/vhost.d:rw
      - /mnt/proxy-files/html:/usr/share/nginx/html:rw
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - ACME_CA_URI=https://acme-staging-v02.api.letsencrypt.org/directory
    restart: unless-stopped

[Other container exist here but do not conflict with pihole and function ok]

  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    ports:
      - '53:53/tcp'
      - '53:53/udp'
      - "67:67/udp"
      - '8080:80/tcp'
      - "8443:443/tcp"
    dns:
      - 127.0.0.1
      - 1.1.1.1
    environment:
      - TZ='Europe/Zurich'
      - WEBPASSWORD="somepassword"
      - VIRTUAL_HOST=[pi hole domain]
      - PROXY_LOCATION=pihole
      - VIRTUAL_PORT=80
      - ServerIP=[Elastic IP pointing to this server, server itself has different IP address]
    depends_on:
      - proxy  
    cap_add:
      - NET_ADMIN
    volumes:
      - /mnt/pi-files/pihole:/etc/pihole
      - /mnt/pi-files/dnsmasq:/etc/dnsmasq.d
    restart: unless-stopped

Now, as soon as I do a netplan apply to change the DNS servers locally, DNS Stops working all together. No resolution is possible.
I confirm that all settings are correct

universe@universe:/opt$ cat /etc/resolv.conf 
nameserver 127.0.0.1
universe@universe:/opt$

universe@universe:~$ nslookup 
> server
Default server: 127.0.0.1
Address: 127.0.0.1#53
> google.com
;; connection timed out; no servers could be reached

I can also see the necessary ports bound

universe@universe:/opt$ docker ps
CONTAINER ID   IMAGE                                    COMMAND                  CREATED          STATUS                      PORTS                                                                                                     NAMES
bc9537611c96   pihole/pihole:latest                     "/s6-init"               38 minutes ago   Up 12 minutes (unhealthy)   0.0.0.0:53->53/udp, 0.0.0.0:53->53/tcp, 0.0.0.0:67->67/udp, 0.0.0.0:8080->80/tcp, 0.0.0.0:8443->443/tcp   pihole
0278f4637483   jrcs/letsencrypt-nginx-proxy-companion   "/bin/bash /app/entr…"   38 minutes ago   Up 12 minutes                                                                                                                         letsencrypt
de675195484d   jwilder/nginx-proxy:alpine               "/app/docker-entrypo…"   38 minutes ago   Up 12 minutes               0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                                  nginx-proxy

I can also see that the upstream pihole container isnt responding to web requests and other DNS resolution requests are failing bringing down the whole system

nginx-proxy    | nginx.1    | pihole.domain  - - [03/Jan/2021:13:02:30 +0000] "GET / HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0"
nginx-proxy    | nginx.1    | 2021/01/03 13:02:44 [error] 63#63: ocsp.stg-int-x1.letsencrypt.org could not be resolved (110: Operation timed out) while requesting certificate status, responder: ocsp.stg-int-x1.letsencrypt.org, certificate: "/etc/nginx/certs/crtfile"
nginx-proxy    | nginx.1    | pihole.domain 159.100.243.64 - - [03/Jan/2021:13:02:50 +0000] "GET / HTTP/1.1" 504 167 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:84.0) Gecko/20100101 Firefox/84.0"
nginx-proxy    | nginx.1    | 2021/01/03 13:02:50 [error] 63#63: *189 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 159.100.243.64, server: pihole.mustafa.lol, request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:80/", host: "pihole.mustafa.lol"

Can you also advise on what PROXY_LOCATION exactly is? There is no documentation on the site here and I have no idea exactly what to put in considering my setup.

Also is there a different user from root that the container uses? If there is, then I need to set the necessary ownership on bind mounted folders but I cant seem to find any documentation here? The docker setup scripts here also dont show it so I am assuming it runs as root?

[muzzah]

Also I think this is causing the set password in the docker compose file not to be set since I cannot log in with the specified password.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Please comment or update this issue or it will be closed in 5 days.