CI: coredns crashloopbackoff in packet_centos7-calico-ha-once-localhost

[VannTen]

Which jobs are failing ?

packet_centos7-calico-ha-once-localhost

Which tests are failing ?

020_check-pods-running

Since when has it been failing ?

Yesterday (February 1st 2024) (first failure : https://gitlab.com/kargo-ci/kubernetes-sigs-kubespray/-/jobs/6073443810 )

Reason for failure

Unknown, but I suspect #10719

Anything else we need to know

No response

[VannTen]

Looks like the issue is that coredns gets a permission denied error when trying to bind to port 53 (if I reproduced the problem correctly). Might also come from docker or centos7

[VannTen]

Possible cause (at least similar symptoms coredns/coredns#6249 )

[pacoxu]

Kubespray is using a very old docker version 20.10.20.

moby/moby#43420 is reported on 20.10.14+ for a similar problem and they fixed it in pi-hole/docker-pi-hole#1030. (Not fixed in moby).

Later moby/moby#45491 another similar problem was fixed and backported to 23.0.7 moby/moby#46222.

Can we upgrade docker to 23.0.7+ to see if it can be fixed?

In moby/moby#46222, docker maintainer provided a simple steps to check if it is fixed.

Use the below as Dockerfile

FROM alpine
RUN apk add --update libcap

RUN ls -la /usr/sbin/capsh
RUN setcap 'cap_sys_admin=ep' /usr/sbin/capsh
docker run --security-opt=no-new-privileges --user=100 --cap-add sys_admin <tag of the build Dockerfile> capsh --print

You should see Current: = cap_sys_admin+ep rather than Current: