Docker working directory changed

[izenn]

Upgraded from 1.10.1 to 1.11.0 and coredns was not reading the config from /Corefile or /hosts anymore

steps to reproduce:
deploy docker image with volume mounts to /Corefile and /hosts

per docker inspect, the working directory for the docker container was changed from / to /home/nonroot

using coredns/coredns:latest from hub.docker.com

[NikVandewouwer]

Not sure if related, but also for me coredns started crashing this morning after pulling latest (1.11.0) due latest tag:

  coredns:
    image: coredns/coredns:latest
    container_name: coredns
    volumes:
      - ${ROOT_DIR}/docker/coredns:/etc/coredns
    command: -conf /etc/coredns/Corefile
    expose:
      - 53
      - 53/udp
    ports:
      - 53:53
      - 53:53/udp
    restart: unless-stopped

I've received continuously the following error in the logs every minute:
Listen: listen tcp :53: bind: permission denied

After reverting back to 1.10.1 everything works again:

  coredns:
    image: coredns/coredns:1.10.1
    container_name: coredns
    volumes:
      - ${ROOT_DIR}/docker/coredns:/etc/coredns
    command: -conf /etc/coredns/Corefile
    expose:
      - 53
      - 53/udp
    ports:
      - 53:53
      - 53:53/udp
    restart: unless-stopped

(I'm using docker-compose on synology nas)

[SuperQ]

I'm guessing this is related to the user ID change.

#5969

[NikVandewouwer]

@SuperQ could you guide me in the right direction what to do to able to run 1.11.0?

[chrisohaver]

@NikVandewouwer, Try adding NET_BIND_SERVICE capability

[chrisohaver]

@NikVandewouwer, Try adding NET_BIND_SERVICE capability

Nevermind - I was thinking this was a privileged port bind issue, not a directory issue. I wasn't paying attention.

[chrisohaver]

@NikVandewouwer, Try adding NET_BIND_SERVICE capability

Nevermind - I was thinking this was a privileged port bind issue, not a directory issue. I wasn't paying attention.

OK, I see there are two issue reported here. Second is a port bind issue: Listen: listen tcp :53: bind: permission denied
Adding NET_BIND_SERVICE capability should resolve that.

[NikVandewouwer]

@chrisohaver how would I update my docker-compose to add NET_BIND_SERVICE ? The documentation I find is mainly around k8s.

This for example leads to the same error

  coredns:
    image: coredns/coredns:latest
    container_name: coredns
    environment:
      - PUID=${UID}
      - PGID=${GID}
    volumes:
      - ${ROOT_DIR}/docker/coredns:/etc/coredns
    command: -conf /etc/coredns/Corefile
    expose:
      - 53
      - 53/udp
    ports:
      - 53:53
      - 53:53/udp
    cap_add:
      - NET_BIND_SERVICE
    restart: unless-stopped

[chrisohaver]

I dont know off hand.
Alternately, you could try binding to an alternate port in the container (eg 5353), mapping it to the exposed 53 port. You’d also need to update corefile to listen to that alternate port.

[davidgiga1993]

This change in the dockerfile bricked coredns for us. The environment is highly restricted and we're binding coredns on a higher port, however due to the mandatory cap_net_bind_service the latest coredns version doesn't start anymore. Allowing this capability isn't an option for us.

Can you please remove that capability from the dockerfile (or create two image variants?). The capability shouldn't be added inside the dockerfile directly.

[SuperQ]

You have a highly restricted environment and you use public container images?

[davidgiga1993]

You have a highly restricted environment and you use public container images?

We mirror them and they are being scanned.. this however has nothing to do with my issue :)

[polarathene]

UPDATE: I have raised a PR removing the setcap change. Unclear at this stage if it would be accepted as it seems the decision may negatively affect one groups needs.

The proper fix would be to have coredns request when binding for privileged ports only when it's actually needed for the non-root user (possibly k8s, and with Docker when using host mode networking). That's a bunch more effort to implement though 🤷‍♂️

---

Allowing this capability isn't an option for us.

Can you please remove that capability from the dockerfile (or create two image variants?). The capability shouldn't be added inside the dockerfile directly.

Could you clarify? Are you running rootless? (I reproduced with a standard install at rootful daemon)

Files:

# compose.yaml
services:
  coredns:
    image: coredns/coredns:latest
    container_name: coredns
    volumes:
      - /tmp/Corefile:/Corefile
    # Restore previous working directory:
    working_dir: /
    # Alternatively direct the command to the custom config location:
    #command: -conf /Corefile

# Corefile
example.test {
  whoami
}

. {
  forward . 8.8.8.8
}

Bring that up with docker compose up --force-recreate and get the IP address assigned to it docker inspect coredns | grep IPAddress with output such as 172.18.0.2.

Now you can query it from the host (or another container on the same container network):

# Working whoami for configured domain:
dig @172.18.0.2 example.test | grep '^example'

# Working forward:
dig @172.18.0.2 google.com | grep '^google'

I can add the ports and observe no problems in the logs for the container (in this case the host already had :53 in use, so I bound to :54 instead):

    ports:
      - 54:53
      - 54:53/udp

Now you can query localhost on port 54:

# Working whoami for configured domain:
dig @127.0.0.1 -p 54 example.test | grep '^example'

# Working forward:
dig @127.0.0.1 -p 54 google.com | grep '^google'

On the test system (Ubuntu 22.04.3 LTS, fresh VM server install), running as a non-root user (but configured for the docker group) there was no issues here requiring capability added, but dropping it would fail:

# docker logs coredns
# exec /coredns: operation not permitted
    cap_drop:
      - NET_BIND_SERVICE

And as expected that's a non-issue if running the 1.10.1 tag instead of the current :latest (1.11.1).

---

Is the setcap really meaningful / required?

Dropping the capability requirement (with setcap cap_net_bind_service=-ep /coredns, because I'm too lazy to do a full build 😅 ), everything seems to work fine even with cap_drop applied and leaving the non-root user.

So the question is what value is that change providing, or am I doing something wrong?

If the user needs to have the capability can't they do so via cap_add? (usually how this is addressed in docker land AFAIK)

What value is provided by enforcing it on the binary within the container? From what I understand the setcap is making the capability requirement explicit, but it doesn't appear required to function, so wouldn't it be wiser to respect the environment?

---

I've received continuously the following error in the logs every minute:
Listen: listen tcp :53: bind: permission denied

(I'm using docker-compose on synology nas)

I'm not sure how to reproduce that, but I guess it's due to the host needing that capability enabled? (or perhaps with your docker / docker compose binary?)

I know that Podman at least runs with less default capabilities. I got a different error as shown above when dropping the capability.

EDIT: I was able to reproduce that on 1.10.1 by running with a non-root user. On 1.11.1 it should fail with a less helpful error when the capability isn't available, otherwise it should bypass the restriction for unprivileged ports (which Docker did remove for internal bridges around 2020Q4).

My guess is it might be due to your kernel as those NAS tend to be quite old, but it could also be the Docker version. I'd add that if you built the image yourself from the Dockerfile that without buildx / DOCKER_BUILDKIT=1 the COPY directive will lose the capabilities applied to the coredns binary. This may also be affected by filesystem / volume driver too 😅

[pacoxu]

Listen: listen tcp :53: bind: permission denied

moby/moby#45491: a similar problem was fixed and backported to docker v23.0.7+ & v24.0.6+ & 25.0.0. moby/moby#46222. For this bind permission denied error, could you check if it is fixed in new docker versions.

See kubernetes-sigs/kubespray#10869 (comment) for detials.

[polarathene]

EDIT: Sorry, below was regarding a related issue that outputs operation not permitted.

---

For this bind permission denied error, could you check if it is fixed in new docker versions.

Last I checked it is not and there is no reason it should be "fixed", as it's doing what is intended.

I was working on an example in late Dec to share but have been side tracked elsewhere unfortunately.

---

If you are security conscious and drop all capabilities that you don't need, the approach CoreDNS has taken conflicts with that.

• Using setcap to set the effective capability bit enforces a kernel check for that capability to be available.
• If you know that you don't need that capability (eg: You want to bind port 5353 instead of 53), but the program is "capability-dumb" only relying on the file attribute from setcap... the binary won't run as it fails the kernel check.
• Instead CoreDNS should at run-time check for the capability in the effective set when it's needed. You'd still use setcap to add the capability to the permitted set, but leave CoreDNS to handle the call that raises it into the effective set.