cis-drivers not working with v2.2.1-k3sv1.26.4-k3s1

[rsheng118]

Kairos version:

NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.18.0
PRETTY_NAME="Alpine Linux v3.18"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
KAIROS_NAME="kairos-alpine-ubuntu"
KAIROS_VERSION="v2.2.1-k3sv1.26.4+k3s1"
KAIROS_ID="kairos"
KAIROS_ID_LIKE="kairos-alpine-ubuntu"
KAIROS_VERSION_ID="v2.2.1-k3sv1.26.4+k3s1"
KAIROS_PRETTY_NAME="kairos-alpine-ubuntu v2.2.1-k3sv1.26.4+k3s1"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues/new/choose"
KAIROS_HOME_URL="https://github.com/kairos-io/provider-kairos"
KAIROS_IMAGE_REPO="quay.io/kairos/kairos-alpine-ubuntu"
KAIROS_IMAGE_LABEL="latest"
KAIROS_GITHUB_REPO="kairos-io/provider-kairos"
KAIROS_VARIANT="core"
KAIROS_FLAVOR="opensuse-leap"

CPU architecture, OS, and Version:
x86, vmware

Describe the bug

with v2.2.0+ and above the csi-drivers is unable to spin up.

also, the v2.2.0+ seems not merge cloud-config files, when i have write_files block in separate files, in v2.1.0, it will combine those two blocks together, but v2.2.1 release seems to just over ride the first block.

            ---
            apiVersion: helm.cattle.io/v1
            kind: HelmChart
            metadata:
              name: csi-driver-nfs
              namespace: kube-system
            spec:
              chart: csi-driver-nfs
              targetNamespace: kube-system
              repo: https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts

  Normal   Pulled          36m (x167 over 76m)  kubelet            Container image "registry.k8s.io/sig-storage/nfsplugin:v4.4.0" already present on machine
  Normal   SandboxChanged  33m                  kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled          33m                  kubelet            Container image "registry.k8s.io/sig-storage/csi-provisioner:v3.5.0" already present on machine
  Normal   Created         33m                  kubelet            Created container csi-provisioner
  Normal   Started         33m                  kubelet            Started container csi-provisioner
  Normal   Pulled          33m                  kubelet            Container image "registry.k8s.io/sig-storage/csi-snapshotter:v6.2.2" already present on machine
  Normal   Created         33m                  kubelet            Created container csi-snapshotter
  Normal   Started         33m                  kubelet            Started container csi-snapshotter
  Normal   Pulled          33m                  kubelet            Container image "registry.k8s.io/sig-storage/livenessprobe:v2.10.0" already present on machine
  Normal   Created         33m                  kubelet            Created container liveness-probe
  Normal   Started         33m                  kubelet            Started container liveness-probe
  Warning  Failed          33m                  kubelet            Error: failed to generate container "bf5dd328876de5a6d36628a479f7cf35382460d77c8c7d6b09249ae0919760ec" spec: failed to generate spec: path "/var/lib/kubelet/pods" is mounted on "/var/lib/kubelet" but it is not a shared mount
  Warning  Failed          33m                  kubelet            Error: failed to generate container "4dcbed390b77eace062e63309e3f64a9c5fd027e746f8525df63a87f725b26ec" spec: failed to generate spec: path "/var/lib/kubelet/pods" is mounted on "/var/lib/kubelet" but it is not a shared mount
  Warning  Failed          33m                  kubelet            Error: failed to generate container "b31a1a29c7c740605ace1255651af2b7ae2518a226d657a813f7e87b584158ae" spec: failed to generate spec: path "/var/lib/kubelet/pods" is mounted on "/var/lib/kubelet" but it is not a shared mount
  Warning  Failed          33m                  kubelet            Error: failed to generate container "53a2eaa84fb80db979390207bd2cc152b959e16d98663cf6a145f60fe7cadc2c" spec: failed to generate spec: path "/var/lib/kubelet/pods" is mounted on "/var/lib/kubelet" but it is not a shared mount
  Warning  Failed          32m                  kubelet            Error: failed to generate container "1bd46d53792510320bc34bc047e80f1cba6e8d154870e2aef3a791272259ceda" spec: failed to generate spec: path "/var/lib/kubelet/pods" is mounted on "/var/lib/kubelet" but it is not a shared mount
  Warning  Failed          32m                  kubelet            Error: failed to generate container "1d7848ee1a164c3494642c2936a695c36efd52bd59981f87046e2e22e9adfd09" spec: failed to generate spec: path "/var/lib/kubelet/pods" is mounted on "/var/lib/kubelet" but it is not a shared mount
  Warning  Failed          32m                  kubelet            Error: failed to generate container "b60cba7bd604b535601f888eee2b344073b67b5041f25670f91e94045cb13457" spec: failed to generate spec: path "/var/lib/kubelet/pods" is mounted on "/var/lib/kubelet" but it is not a shared mount

reverted back to v2.1.0-k3sv1.26.4-k3s1 fixed this issue

To Reproduce
upgrade to v2.2.1-k3sv1.26.4-k3s1

Expected behavior
csi-drivers in running state

Logs

Additional context

[mudler]

Moving on community supported and adding help wanted label as Alpine is a community flavor and not going through Kairos automated QA tests.

https://kairos.io/docs/reference/image_matrix/

[mauromorales]

For anyone interested to help with this, please first check if it's working on the latest release (2.4 atm) and also check other flavors and report.

[kaiehrhardt]

Same for:

NAME="openSUSE Leap"
VERSION="15.5"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.5"
PRETTY_NAME="openSUSE Leap 15.5"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.5"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Leap"
LOGO="distributor-logo-Leap"
KAIROS_PRETTY_NAME="kairos-standard-opensuse-leap-15.5 v2.5.0-v1.28.5-k3s1"
KAIROS_TARGETARCH="arm64"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_NAME="kairos-standard-opensuse-leap-15.5"
KAIROS_VERSION="v2.5.0-v1.28.5-k3s1"
KAIROS_ID_LIKE="kairos-standard-opensuse-leap-15.5"
KAIROS_IMAGE_LABEL="leap-15.5-standard-arm64-rpi4-v2.5.0-k3sv1.28.5-k3s1"
KAIROS_FLAVOR="opensuse"
KAIROS_VARIANT="standard"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_SOFTWARE_VERSION="v1.28.5+k3s1"
KAIROS_ID="kairos"
KAIROS_VERSION_ID="v2.5.0-v1.28.5-k3s1"
KAIROS_FLAVOR_RELEASE="leap-15.5"
KAIROS_MODEL="rpi4"
KAIROS_RELEASE="v2.5.0"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_IMAGE_REPO="quay.io/kairos/opensuse:leap-15.5-standard-arm64-rpi4-v2.5.0-k3sv1.28.5-k3s1"
KAIROS_ARTIFACT="kairos-opensuse-leap-15.5-standard-arm64-rpi4-v2.5.0-k3sv1.28.5+k3s1"

bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.

Sounds like nfs-common is missing. I'm not familiar with opensuse, but could we solve this, if we isntall yast2-nfs-common?

[kaiehrhardt]

PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
KAIROS_VERSION="v2.5.0-v1.28.5-k3s1"
KAIROS_VERSION_ID="v2.5.0-v1.28.5-k3s1"
KAIROS_IMAGE_REPO="quay.io/kairos/ubuntu:22.04-standard-arm64-rpi4-v2.5.0-k3sv1.28.5-k3s1"
KAIROS_FLAVOR_RELEASE="22.04"
KAIROS_VARIANT="standard"
KAIROS_MODEL="rpi4"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_NAME="kairos-standard-ubuntu-22.04"
KAIROS_ID_LIKE="kairos-standard-ubuntu-22.04"
KAIROS_ID="kairos"
KAIROS_IMAGE_LABEL="22.04-standard-arm64-rpi4-v2.5.0-k3sv1.28.5-k3s1"
KAIROS_ARTIFACT="kairos-ubuntu-22.04-standard-arm64-rpi4-v2.5.0-k3sv1.28.5+k3s1"
KAIROS_TARGETARCH="arm64"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_SOFTWARE_VERSION="v1.28.5+k3s1"
KAIROS_PRETTY_NAME="kairos-standard-ubuntu-22.04 v2.5.0-v1.28.5-k3s1"
KAIROS_RELEASE="v2.5.0"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_FLAVOR="ubuntu"

$ apt list | grep nfs-common
nfs-common/jammy-updates 1:2.6.1-1ubuntu1.2 arm64

ls /usr/sbin/ | grep mount
mount.fuse
mount.fuse3

docker run -it --rm quay.io/kairos/ubuntu:22.04-standard-amd64-generic-v2.5.0-k3sv1.27.9-k3s1 sh
ls /usr/sbin | grep mount
mount.fuse
mount.fuse3
mount.nfs
mount.nfs4
mount.zfs
mountstats
showmount
umount.nfs
umount.nfs4

Any idea @mauromorales @mudler?