Failing to mount cephfs volumes with invalid argument

[Starttoaster]

Describe the bug

I upgraded ceph-csi in my clusters to 3.14.0 last week, for both rbd and cephfs. I began seeing cephfs volumes failing to mount with an error from nodeplugin stating invalid argument.

I'm not sure what the invalid argument actually is, the logs don't quite make that clear. But I noted this old Issue exhibited a similar logline #2254

And I noted in the 3.14.0 release this PR #5090 changed the cephfs mount syntax. Not sure if it's related, but seemed suspicious.

Environment details

• Image/version of Ceph CSI driver : 3.14.0
• Helm chart version : 3.14.0
• Kernel version : 5.15.0-135-generic on the k8s nodes
• Mounter used for mounting PVC (for cephFS its fuse or kernel. for rbd its
  krbd or rbd-nbd) : we didn't specify it, so admittedly I'm not sure. I'll update this when I find out
• Kubernetes cluster version : 1.32.3
• Ceph cluster version : 18.2.2

Steps to reproduce

Steps to reproduce the behavior:

1. Setup details: 'Deploy the latest release (3.14.0) of ceph-csi for cephfs with mostly default settings', values file below:

csiConfig:
  - clusterID: $CEPH_CLUSTER_ID
    monitors: $CEPH_MONITORS
    readAffinity:
      enabled: true
      crushLocationLabels:
        - topology.kubernetes.io/region
        - topology.kubernetes.io/zone

readAffinity:
  enabled: true
  crushLocationLabels:
    - topology.kubernetes.io/region
    - topology.kubernetes.io/zone

nodeplugin:
  httpMetrics:
    containerPort: 8187

storageClass:
  create: true
  name: cephfs-nvme
  clusterID: $CEPH_CLUSTER_ID
  fsName: $CEPHFS_FS
  pool: $CEPHFS_POOL

secret:
  create: true
  adminID: $CEPHFS_USERNAME
  adminKey: $CEPHFS_KEY

kubeletDir: /var/lib/k0s/kubelet

2. Create a PVC with ReadWriteMany, and a Pod that mounts it.
3. See error

Actual results

ceph-csi failed to mount these volumes, receiving an invalid argument error.

Expected behavior

cephfs would mount my volumes.

Logs

I started seeing some of my Pods that mount cephfs volumes failing to start. The event on the Pod says MountVolume.MountDevice failed for volume "pvc-28619aba-591a-4ba1-9288-17c9e266b5cc" : rpc error: code = Internal desc = rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-22, Invalid argument

I viewed my ceph-csi logs for errors, and saw the following error show up: ID: 11448 Req-ID: 0001-0024-7e7d8f94-38e4-454e-9f86-caf6f2708cd9-0000000000000002-4d521645-f903-440c-b501-bd6b672dfaa1 GRPC error: rpc error: code = Internal desc = rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-22, Invalid argument

That ceph-csi log comes from the nodeplugin component in the csi-cephfsplugin container.

Additional context

The arguments given to the csi-cephfsplugin container are: --nodeid=$(NODE_ID) --type=cephfs --nodeserver=true --pidlimit=-1 --kernelmountoptions= --fusemountoptions= --endpoint=$(CSI_ENDPOINT) --v=5 --drivername=$(DRIVER_NAME) --enable-read-affinity=true --crush-location-labels=topology.kubernetes.io/region,topology.kubernetes.io/zone --logslowopinterval=30s

I noticed that there are no kernelmountoptions or fusemountoptions listed. Not sure if that's important. We just updated from 3.13.1 to 3.14.0 without any values file changes and began seeing this issue.

For example:

Any existing bug report which describe about the similar issue/behavior

[Starttoaster]

I just downgraded all my clusters back to ceph-csi 3.13.1 and my Pods that were stuck waiting for the volume to mount started right up again.

[Starttoaster]

I maybe see what the issue is. We are using the admin keys from the auth Secret that were removed a few months back 890af07

[Starttoaster]

So the solution for us for this was in the helm vars for the auth Secret:

secret:
  create: true
- adminID: $CEPHFS_USERNAME
- adminKey: $CEPHFS_KEY
+ userID: $CEPHFS_USERNAME
+ userKey: $CEPHFS_KEY

For us in particular, we were always using non-admin credentials in the adminID and adminKey fields, so I didn't have to change the values at all, just the fields' keys.

I was curious, should it be expected that breaking changes are put into minor releases of ceph-csi? What are the rules that ceph-csi is using for semver version changes? And in the future, could a breaking change like this one be called out as a breaking change in the release's changelog?

Additionally, I'm not sure how simple of a feat this is, but noting that the person from 2021 had a similar issue with their auth secret being incorrect -- maybe logging could be improved in the case that invalid credentials were passed to ceph-csi for ceph. invalid argument could mean basically anything..

[black-dragon74]

For us in particular, we were always using non-admin credentials in the adminID and adminKey fields, so I didn't have to change the values at all, just the fields' keys.

That is correct. We have already been supporting user credentials in CephFS. With the above change we just removed the admin* keys.

I was curious, should it be expected that breaking changes are put into minor releases of ceph-csi?

Not ideally. While it was a minor change, I acknowledge your point in it being highlighted in the release notes. We did not remove support for admin* in CephFS secrets but rather deprecated them. See this for more details.

maybe logging could be improved in the case that invalid credentials were passed to ceph-csi for ceph.

While using admin* credentials, a deprecation warning is logged. The invalid argument comes from librados, ceph-csi just bubbles it up.

[Starttoaster]

Interesting, you're saying that the admin credentials in the auth secret were still respected, just deprecated? And that seems at odds with what I experienced where using the admin* credentials actually led to none of my cephfs volumes being mounted anymore until I either downgraded to a previous release, or used the new auth secret user ID and key fields.

I also didn't get that logline you referenced about switching to the new keys fields in the auth secret. The only logline I had that signaled an issue within ceph-csi was the invalid argument log I mentioned. Which, sure, comes from librados, but I guess then I'm curious why I never got the more useful log from your code that calls out the admin credentials :) That log would have been way more useful.

[black-dragon74]

Interesting, you're saying that the admin credentials in the auth secret were still respected, just deprecated? And that seems at odds with what I experienced where using the admin* credentials actually led to none of my cephfs volumes being mounted anymore until I either downgraded to a previous release, or used the new auth secret user ID and key fields.

The reason it failed for you is because helm was unable to substitute the required credentials in k8s secrets as it was looking for user* in Values.secrets which led to empty secrets being created in the cluster (where userID and userKey were present with value as ""), hence the invalid argument.

I'm curious why I never got the more useful log from your code that calls out the admin credentials

That log message will be encountered only when k8s secrets are created with admin* field keys, which was not true in our case (the secrets were created with expected field keys i.e. user*, it's just that the values for those fields were empty).

If you (really) want to see that log, edit the secret that is created by helm manually and update the field keys from user* to admin*, everything will keep working the same.

Hope that explains it :)

P.S: We have a fix in progress where missing secret.user* from values.yaml will be treated as an error.

Regards

[Starttoaster]

Thanks for the update and clarification! For what it's worth, our auth secret had both the admin and user fields in it. But it sounds like only the user fields, which were empty in our case, were being respected in it.

Anyway, I appreciate this getting better error messaging, and future breaking changes requiring being called out in the changelog for a release (and preferably be found only in major releases) :)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.