Mounts permission denied starting in 33.20210426.3.0

[dghubble]

Describe the bug

Kubernetes Pod workloads on nodes starting in 33.20210426.3.0 now see permission denied errors accessing mounts.

From my own collection of clusters (mix of clouds, FCOS/Flatcar, channels, configurations), I observed this with new clusters created today having all nodes NotReady (flannel, Calico, and Cilium all need to write plugins to a host mount) and other workloads having denials accessing configmaps. Existing clusters on older releases or new clusters that intentionally use an old release were ok.

# flannel
Writing CNI binaries to host /opt/cni/bin
cp: can't create '/host//opt/cni/bin/bandwidth': Permission denied
cp: can't create '/host//opt/cni/bin/bridge': Permission denied
cp: can't create '/host//opt/cni/bin/flannel': Permission denied
cp: can't create '/host//opt/cni/bin/host-local': Permission denied
cp: can't create '/host//opt/cni/bin/loopback': Permission denied
cp: can't create '/host//opt/cni/bin/portmap': Permission denied

# calico
kubectl logs calico-node-pvhrs -n kube-system -c install-cni
time="2021-05-05T19:55:53Z" level=info msg="Running as a Kubernetes pod" source="install.go:140"
time="2021-05-05T19:55:53Z" level=info msg="/host/opt/cni/bin is not writeable, skipping"
time="2021-05-05T19:55:53Z" level=info msg="/host/secondary-bin-dir is not writeable, skipping"
time="2021-05-05T19:55:53Z" level=fatal msg="open /host/etc/cni/net.d/calico-kubeconfig: permission denied" source="install.go:472"

# cilium
unable to read configuration directory: open /tmp/cilium/config-map: permission denied" subsys=config

# coredns
loading Caddyfile via flag: open /etc/coredns/Corefile: permission denied

When

For the stable channel, the issue appears in 33.20210426.3.0 (released yesterday).

Nodes with earlier release 33.20210412.3.0 (both existing and new ones created today) work as expected.

Scope

Likely across platforms. I've checked AWS and DigitalOcean myself.

Workaround

You can set CNI pods to run with privileged (acts as spc_r). This allows other pods to start, but any that need to read configmaps or other mounts will see permission denied errors. Effectively you cannot get a working cluster with the new OS image.

Reproduction steps

Steps to reproduce the behavior:

1. Create a Kubernetes cluster with any FCOS channel (uses latest release from that channel)
2. Observe pods see permission denied accessing data on mounts

System details

• Bare Metal/QEMU/AWS/GCP/etc.
• Fedora CoreOS version 33.20210426.3.0
• Runtime: docker-shim

Additional information

Examples of DaemonSet manifest host mounts.

• https://github.com/poseidon/terraform-render-bootstrap/blob/master/resources/flannel/daemonset.yaml#L46
• https://github.com/poseidon/terraform-render-bootstrap/blob/master/resources/calico/daemonset.yaml#L67

[dghubble]

The OS change in question had 45 package updates. https://getfedora.org/en/coreos?stream=stable

Some new observations narrow this to docker or SELinux related packages:

• Kubernetes clusters using the default docker-shim runtime are affected
• Kubernetes clusters using the containerd runtime (something I've been testing) are not affected

EDIT: typo

[basvdlei]

I ran into a similar issue, this was due to a change in podman no longer relabeling (:z) volumes for privileged containers: containers/podman#10209

Sadly we did not catch it in testing because the volumes of upgraded nodes were already relabeled by the previous version of podman. Only when trying to (re)build a node this became an issue.

As a work-around, I've added manual relabeling in the systemd unit:

ExecStartPre=/bin/mkdir -p /var/lib/example
ExecStartPre=/bin/chcon -R -t container_file_t -l s0 /var/lib/example

[dustymabe]

For all the permission denied errors, are there related SELinux denials in the journal?

• Kubernetes clusters using the default docker-shim runtime are affected
• Kubernetes clusters using the containerd runtime (something I've been testing) are not unaffected

Just for clarity.. Do you mean containerd runtime clusters are not affected rather than are not unaffected?

[dustymabe]

Here's the set of packages that changed:

ostree diff commit from: 8563aca63f27acdc80c7104486bbc50435dfa47eff28dc7e9209f2496b166076
ostree diff commit to:   49ec34ca76b71283d317025a54fe815422299ced4d035369599539e2665d5818
Upgraded:
  NetworkManager 1:1.26.6-1.fc33 -> 1:1.26.8-1.fc33
  NetworkManager-cloud-setup 1:1.26.6-1.fc33 -> 1:1.26.8-1.fc33
  NetworkManager-libnm 1:1.26.6-1.fc33 -> 1:1.26.8-1.fc33
  NetworkManager-team 1:1.26.6-1.fc33 -> 1:1.26.8-1.fc33
  NetworkManager-tui 1:1.26.6-1.fc33 -> 1:1.26.8-1.fc33
  clevis 16-2.fc33 -> 18-1.fc33
  clevis-dracut 16-2.fc33 -> 18-1.fc33
  clevis-luks 16-2.fc33 -> 18-1.fc33
  clevis-systemd 16-2.fc33 -> 18-1.fc33
  conmon 2:2.0.26-1.fc33 -> 2:2.0.27-2.fc33
  containers-common 4:1-9.fc33 -> 4:1-15.fc33
  crun 0.18-5.fc33 -> 0.19.1-2.fc33
  cups-libs 1:2.3.3op2-3.fc33 -> 1:2.3.3op2-4.fc33
  dnsmasq 2.84-1.fc33 -> 2.85-1.fc33
  fedora-release-common 33-3 -> 33-4
  fedora-release-coreos 33-3 -> 33-4
  fedora-release-identity-coreos 33-3 -> 33-4
  flatpak-session-helper 1.10.2-2.fc33 -> 1.10.2-3.fc33
  fstrm 0.6.0-1.fc33 -> 0.6.1-2.fc33
  fwupd 1.5.8-1.fc33 -> 1.5.9-2.fc33
  grub2-common 1:2.06~rc1-1.fc33 -> 1:2.06~rc1-2.fc33
  grub2-efi-x64 1:2.06~rc1-1.fc33 -> 1:2.06~rc1-2.fc33
  grub2-pc 1:2.06~rc1-1.fc33 -> 1:2.06~rc1-2.fc33
  grub2-pc-modules 1:2.06~rc1-1.fc33 -> 1:2.06~rc1-2.fc33
  grub2-tools 1:2.06~rc1-1.fc33 -> 1:2.06~rc1-2.fc33
  grub2-tools-minimal 1:2.06~rc1-1.fc33 -> 1:2.06~rc1-2.fc33
  kernel 5.10.19-200.fc33 -> 5.11.15-200.fc33
  kernel-core 5.10.19-200.fc33 -> 5.11.15-200.fc33
  kernel-modules 5.10.19-200.fc33 -> 5.11.15-200.fc33
  libkcapi 1.2.0-3.fc33 -> 1.2.1-1.fc33
  libkcapi-hmaccalc 1.2.0-3.fc33 -> 1.2.1-1.fc33
  libluksmeta 9-8.fc33 -> 9-9.fc33
  libsmbclient 2:4.13.7-0.fc33 -> 2:4.13.7-1.fc33
  libtirpc 1.2.6-2.rc4.fc33 -> 1.2.6-3.rc4.fc33
  libwbclient 2:4.13.7-0.fc33 -> 2:4.13.7-1.fc33
  libxcrypt 4.4.18-1.fc33 -> 4.4.19-1.fc33
  luksmeta 9-8.fc33 -> 9-9.fc33
  podman 2:3.1.0-2.fc33 -> 2:3.1.2-1.fc33
  podman-plugins 2:3.1.0-2.fc33 -> 2:3.1.2-1.fc33
  runc 2:1.0.0-375.dev.git12644e6.fc33 -> 2:1.0.0-377.rc93.fc33
  samba-client-libs 2:4.13.7-0.fc33 -> 2:4.13.7-1.fc33
  samba-common 2:4.13.7-0.fc33 -> 2:4.13.7-1.fc33
  samba-common-libs 2:4.13.7-0.fc33 -> 2:4.13.7-1.fc33
  samba-libs 2:4.13.7-0.fc33 -> 2:4.13.7-1.fc33
  vim-minimal 2:8.2.2735-1.fc33 -> 2:8.2.2787-1.fc33

The container related updates I see are:

  kernel 5.10.19-200.fc33 -> 5.11.15-200.fc33
  crun 0.18-5.fc33 -> 0.19.1-2.fc33
  containers-common 4:1-9.fc33 -> 4:1-15.fc33
  podman 2:3.1.0-2.fc33 -> 2:3.1.2-1.fc33
  podman-plugins 2:3.1.0-2.fc33 -> 2:3.1.2-1.fc33
  runc 2:1.0.0-375.dev.git12644e6.fc33 -> 2:1.0.0-377.rc93.fc33

I don't think this is podman related since I doubt k8s clusters that use docker-shim leverage podman at all. I don't think crun is used by docker at all.

So that would leave kernel/containers-common/runc.

[dustymabe]

@dghubble can you try out testing-devel build 33.20210423.20.2 (ami-0480ed83b97d0f093) and report if it works or not. Right after that is when the bulk of the container related updates (mentioned above) landed.

[basvdlei]

I don't think this is podman related since I doubt k8s clusters that use docker-shim leverage podman at all. I don't think crun is used by docker at all.

The policies for Podman and Docker do share access to the same container_file_t label. So volumes relabeled by a privileged podman system container (eg. kubelet) previously allowed the docker containers from the pods to access these volumes.

Looks like Typhoon is doing exactly this: https://github.com/poseidon/typhoon/blob/bc9644371017921fb484c1317026ca25a8734180/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml#L39-L43

[dustymabe]

I don't think this is podman related since I doubt k8s clusters that use docker-shim leverage podman at all. I don't think crun is used by docker at all.

The policies for Podman and Docker do share access to the same container_file_t label. So volumes relabeled by a privileged podman system container (eg. kubelet) previously allowed the docker containers from the pods to access these volumes.

Looks like Typhoon is doing exactly this: https://github.com/poseidon/typhoon/blob/bc9644371017921fb484c1317026ca25a8734180/aws/fedora-coreos/kubernetes/workers/fcc/worker.yaml#L39-L43

Ahh - didn't realize they were using podman to run the kubelet. In that case, you're probably right.

That testing-devel I mention in #818 (comment) is before the podman bump, so still useful to get that datapoint.

[dghubble]

@dustymabe yeah I meant using docker-shim runtime is affected and containerd is not affected (testing that configuration is part of why I didn't notice this break).

Thanks for mentioning the podman issue containers/podman#10209 @basvdlei. On-host units like etcd and Kubelet use podman, and :z relabels are needed for these reasons. I'm not sure how using containerd seems to avoid this, but its not a configuration I've PR'd yet anyway.

I'll checkout the AMI mentioned.

[dghubble]

Using testing-devel build 33.20210423.20.2 (ami-0480ed83b97d0f093) does NOT have the problem. Also, looking at the podman change, using :z is intentional, its surprising it would become ignored now.

[rhatdan]

You also run:

podman run -v /var/lib/example:/mnt:z -rm $IMAGE exit 0

And get the same fix.

[dustymabe]

This is a podman bug that has been identified in our stable stream as of 33.20210426.3.0. More details on the bug can be found in containers/podman#10209. It has been fixed upstream in containers/podman#10253. While we wait for that fix to land you can workaround the issue by either setting the context for relevant directories via chcon:

chcon -R -t container_file_t -l s0 /var/lib/example

or by executing a separate dummy podman command without --privileged or --security-opt label=disable before you start launching contianers:

podman run -v /var/lib/example:/mnt:z -rm $IMAGE exit 0

[dustymabe]

NOTE: as @basvdlei mentioned in #818 (comment) this is not as observable on upgraded nodes since older versions of podman would have correctly labeled things when the volumes were first used. You're more likely to see this when deploying new nodes/clusters.

[dghubble]

Thanks all!

[dghubble]

Is this likely to make it into an OS image this week? I need to decide about adding a temporary chcon script in the Kuberntes distro or waiting, since right now users can't choose any channel.

[dghubble]

I tested FCOS testing 34.20210427.2.1, looks good!

[dustymabe]

The fix for this went into next stream release 34.20210503.1.1. Please try out the new release and report issues.

The fix for this went into testing stream release 34.20210427.2.1. Please try out the new release and report issues.

[dustymabe]

Is this likely to make it into an OS image this week? I need to decide about adding a temporary chcon script in the Kuberntes distro or waiting, since right now users can't choose any channel.

@dghubble - it will land in stable next week. That OK?

[dghubble]

Yep, thanks for getting this in 👍

[dustymabe]

The fix for this went into stable stream release 34.20210427.3.0.