rook-ceph-exporter will not start because /var/lib/rook/rook-ceph is not a directory.

[reitermarkus]

I have a rook-ceph-exporter pod stuck in Init:0/1. kubectl describe shows

  Normal   Scheduled    13h                  default-scheduler  Successfully assigned rook-ceph/rook-ceph-exporter-talos-htpc-7fdfb9ffdd-vmfxb to talos-htpc
  Warning  FailedMount  156m (x48 over 13h)  kubelet            Unable to attach or mount volumes: unmounted volumes=[ceph-conf-dir], unattached volumes=[rook-ceph-admin-keyring rook-config-override ceph-daemons-sock-dir rook-ceph-log rook-ceph-crash ceph-conf-dir kube-api-access-bpz6q]: timed out waiting for the condition
  Warning  FailedMount  136m (x48 over 12h)  kubelet            Unable to attach or mount volumes: unmounted volumes=[ceph-conf-dir], unattached volumes=[rook-config-override ceph-daemons-sock-dir rook-ceph-log rook-ceph-crash ceph-conf-dir kube-api-access-bpz6q rook-ceph-admin-keyring]: timed out waiting for the condition
  Warning  FailedMount  61m (x46 over 13h)   kubelet            Unable to attach or mount volumes: unmounted volumes=[ceph-conf-dir], unattached volumes=[rook-ceph-crash ceph-conf-dir kube-api-access-bpz6q rook-ceph-admin-keyring rook-config-override ceph-daemons-sock-dir rook-ceph-log]: timed out waiting for the condition
  Warning  FailedMount  36m (x47 over 13h)   kubelet            Unable to attach or mount volumes: unmounted volumes=[ceph-conf-dir], unattached volumes=[ceph-conf-dir kube-api-access-bpz6q rook-ceph-admin-keyring rook-config-override ceph-daemons-sock-dir rook-ceph-log rook-ceph-crash]: timed out waiting for the condition
  Warning  FailedMount  20m (x53 over 13h)   kubelet            Unable to attach or mount volumes: unmounted volumes=[ceph-conf-dir], unattached volumes=[rook-ceph-log rook-ceph-crash ceph-conf-dir kube-api-access-bpz6q rook-ceph-admin-keyring rook-config-override ceph-daemons-sock-dir]: timed out waiting for the condition
  Warning  FailedMount  15m (x51 over 13h)   kubelet            Unable to attach or mount volumes: unmounted volumes=[ceph-conf-dir], unattached volumes=[ceph-daemons-sock-dir rook-ceph-log rook-ceph-crash ceph-conf-dir kube-api-access-bpz6q rook-ceph-admin-keyring rook-config-override]: timed out waiting for the condition
  Warning  FailedMount  11m (x49 over 13h)   kubelet            Unable to attach or mount volumes: unmounted volumes=[ceph-conf-dir], unattached volumes=[kube-api-access-bpz6q rook-ceph-admin-keyring rook-config-override ceph-daemons-sock-dir rook-ceph-log rook-ceph-crash ceph-conf-dir]: timed out waiting for the condition
  Warning  FailedMount  91s (x416 over 13h)  kubelet            MountVolume.SetUp failed for volume "ceph-conf-dir" : hostPath type check failed: /var/lib/rook/rook-ceph is not a directory

talosctl -n talos-htpc ls -l /var/lib/rook/rook-ceph shows

NODE         MODE         UID   GID   SIZE(B)   LASTMOD           NAME
talos-htpc   drwxr-xr-x   0     0     30        Mar 13 03:24:26   .
talos-htpc   drwxr-xr-x   167   167   20        Mar 13 03:24:26   crash
talos-htpc   drwxr-xr-x   167   167   78        Mar 14 01:07:21   log

So clearly the directory exists.

[reitermarkus]

kubectl describe also shows

Volumes:
  rook-config-override:
    Type:               Projected (a volume that contains injected data from multiple sources)
    ConfigMapName:      rook-config-override
    ConfigMapOptional:  <nil>
  ceph-daemons-sock-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rook/exporter
    HostPathType:  DirectoryOrCreate
  rook-ceph-log:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rook/rook-ceph/log
    HostPathType:  
  rook-ceph-crash:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rook/rook-ceph/crash
    HostPathType:  
  ceph-conf-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rook/rook-ceph
    HostPathType:  Directory
  rook-ceph-admin-keyring:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rook-ceph-admin-keyring
    Optional:    false
  kube-api-access-qjbdx:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true

HostPathType for crash and log seems to be set to an empty string, maybe this is what is causing this.

I manually edited the deployment to change hostPath.type for all directories to DirectoryOrCreate and it seems to have fixed the problem.

[travisn]

@reitermarkus What version of rook are you running? This sounds like an issue from v1.11.0, which has now been fixed in v1.11.1 with #11848. If you're on v1.11.1, @avanthakkar will need to take a look.

[reitermarkus]

I updated from v1.10.11 directly to v1.11.1.

[travisn]

I converted this to an issue in #11910, @avanthakkar PTAL.