scos: ext.config.shared.chrony.dhcp-propagation failed

[HuijingHei]

Aug 30 09:04:55 qemu0 kola-runext-dhcp-propagation[1857]: + podman build -t dnsmasq .
Aug 30 09:04:55 qemu0 kola-runext-dhcp-propagation[1881]: STEP 1/4: FROM registry.fedoraproject.org/fedora:36
Aug 30 09:04:55 qemu0 kola-runext-dhcp-propagation[1881]: Trying to pull registry.fedoraproject.org/fedora:36...
Aug 30 09:04:55 qemu0 kola-runext-dhcp-propagation[1881]: Getting image source signatures
Aug 30 09:04:55 qemu0 kola-runext-dhcp-propagation[1881]: Copying blob sha256:62946078034b7fe37984579d9b82ccf20cc98ffcd6517cf79ffad18e06fe2b23
Aug 30 09:04:55 qemu0 kola-runext-dhcp-propagation[1881]: Copying blob sha256:62946078034b7fe37984579d9b82ccf20cc98ffcd6517cf79ffad18e06fe2b23
Aug 30 09:05:03 qemu0 kola-runext-dhcp-propagation[1881]: Copying config sha256:2ecb6df959942dd2fdeb65606ca2e42a54f8c06af10eeb594fdfc3e2656c53d1
Aug 30 09:05:03 qemu0 kola-runext-dhcp-propagation[1881]: Writing manifest to image destination
Aug 30 09:05:03 qemu0 kola-runext-dhcp-propagation[1881]: Storing signatures
Aug 30 09:05:03 qemu0 kola-runext-dhcp-propagation[1881]: STEP 2/4: RUN dnf -y install systemd dnsmasq iproute iputils && dnf clean all && systemctl enable dnsmasq
Aug 30 09:05:05 qemu0 kola-runext-dhcp-propagation[1925]: Fedora 36 - x86_64                              0.0  B/s |   0  B     00:00
Aug 30 09:05:05 qemu0 kola-runext-dhcp-propagation[1925]: Errors during downloading metadata for repository 'fedora':
Aug 30 09:05:05 qemu0 kola-runext-dhcp-propagation[1925]:   - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64 [getaddrinfo() thread failed to start]
Aug 30 09:05:05 qemu0 kola-runext-dhcp-propagation[1925]: Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64 [getaddrinfo() thread failed to start]
Aug 30 09:05:05 qemu0 kola-runext-dhcp-propagation[1881]: Error: error building at STEP "RUN dnf -y install systemd dnsmasq iproute iputils && dnf clean all && systemctl enable dnsmasq": error while running runtime: exit status 1

[HuijingHei]

Run locally can reproduce the issue, if start container manually with podman, no such issue.

[root@cosa-devsh tmp.HSzl2zlg9Z]# podman build -t dnsmasq .
STEP 1/4: FROM registry.fedoraproject.org/fedora:36
STEP 2/4: RUN dnf -y install systemd dnsmasq iproute iputils && dnf clean all && systemctl enable dnsmasq
Fedora 36 - x86_64                              0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'fedora':
  - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64 [getaddrinfo() thread failed to start]
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64 [getaddrinfo() thread failed to start]
Error: error building at STEP "RUN dnf -y install systemd dnsmasq iproute iputils && dnf clean all && systemctl enable dnsmasq": error while running runtime: exit status 1


[core@cosa-devsh ~]$ sudo podman run -it --rm --name test registry.fedoraproject.org/fedora:36 dnf -y install systemd dnsmasq iproute iputils
...

Installed:
  cryptsetup-libs-2.4.3-2.fc36.x86_64                  dbus-1:1.14.0-1.fc36.x86_64                           dbus-broker-32-1.fc36.x86_64                    
  dbus-common-1:1.14.0-1.fc36.noarch                   dbus-libs-1:1.14.0-1.fc36.x86_64                      device-mapper-1.02.175-7.fc36.x86_64            
  device-mapper-libs-1.02.175-7.fc36.x86_64            diffutils-3.8-2.fc36.x86_64                           dnsmasq-2.86-10.fc36.x86_64                     
  iproute-5.15.0-2.fc36.x86_64                         iputils-20211215-2.fc36.x86_64                        kmod-libs-29-7.fc36.x86_64                      
  libargon2-20171227-9.fc36.x86_64                     libbpf-2:0.7.0-3.fc36.x86_64                          libfdisk-2.38-1.fc36.x86_64                     
  libmnl-1.0.4-15.fc36.x86_64                          libnetfilter_conntrack-1.0.8-4.fc36.x86_64            libnfnetlink-1.0.1-21.fc36.x86_64               
  libseccomp-2.5.3-2.fc36.x86_64                       libxkbcommon-1.4.0-1.fc36.x86_64                      psmisc-23.4-3.fc36.x86_64                       
  qrencode-libs-4.1.1-2.fc36.x86_64                    systemd-250.8-1.fc36.x86_64                           systemd-networkd-250.8-1.fc36.x86_64            
  systemd-pam-250.8-1.fc36.x86_64                      systemd-resolved-250.8-1.fc36.x86_64                  xkeyboard-config-2.35.1-1.fc36.noarch           

Complete!

[cgwalters]

Ah right, so the host system is RHEL8 here, which looks like https://bugzilla.redhat.com/show_bug.cgi?id=2009047

Since RHCOS is cherry picking the podman stack, I suspect we are missing errata.

[cgwalters]

A workaround here for now is likely podman build --security-opt seccomp=unconfined

[HuijingHei]

Thanks @cgwalters , podman build --security-opt seccomp=unconfined works on c9s.
Should I contact container team to fix this? Or just add workaround to https://github.com/coreos/fedora-coreos-config/blob/testing-devel/tests/kola/chrony/dhcp-propagation?

[core@cosa-devsh ~]$ rpm -q podman kernel runc crun 
podman-4.1.1-6.el9.x86_64
kernel-5.14.0-148.el9.x86_64
runc-1.1.3-2.el9.x86_64
crun-1.5-1.el9.x86_64

[cgwalters]

Yeah it's very surprising that we have different seccomp policies applied to build versus run. That seems like a podman bug for sure.

There's a newer podman-4.2.0-3.el9.x86_64.rpm in the latest compose https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/AppStream/x86_64/os/Packages/

[HuijingHei]

Thanks @cgwalters! I will test with that to see if the issue is gone

[HuijingHei]

Test with podman-4.2.0-3.el9.x86_64 and runc-1.1.4-1.el9.x86_64, can reproduce the issue. I will contact to container team about this.

[HuijingHei]

Talk with container team @ypu, file bug https://bugzilla.redhat.com/show_bug.cgi?id=2123246 to track the issue.

Debug with "--log-level debug", the root cause is podman build load config SeccompProfilePath:"/etc/crio/seccomp.json" from cri-o, but podman run works as it load config /usr/share/containers/seccomp.json from containers-common.

The issue can not reproduced on RHEL9.1 as it will default not install cri-o.

[HuijingHei]

For the podman build and run load different config issue is tracked by https://bugzilla.redhat.com/show_bug.cgi?id=2123251

[cgwalters]

Debug with "--log-level debug", the root cause is podman build load config SeccompProfilePath:"/etc/crio/seccomp.json" from cri-o, but podman run works as it load config /usr/share/containers/seccomp.json from containers-common.

OHHHH wow. That's a pretty amazing bug.

The issue can not reproduced on RHEL9.1 as it will default not install cri-o.

OK yeah, this explains why it wasn't found earlier.

So for now...well, hmmm...I think I'd say let's globally add the workaround but with a reminder to ourselves to remove it at some point.

Really this failure has nothing to do with chrony, it's simply podman build is buggy on RHCOS, right? It may make sense to have an explicit separate test for that that we add and denylist to remind ourselves to get it fixed later.

podman build on RHCOS isn't really a major customer-facing thing, but it should work because it will be useful for people debugging things, etc.