failed: lsetxattr read-only file system on pod start #145
Labels
help-wanted
Extra attention is needed
Comments
|
Hey @woehrl01 could you share a minimal manifest that could help reproduce the issue? |
|
Thanks you @mbtamuli Absolutely! Actually I'm using all defaults. The driver is installed with default helm values (1.1.0 and containerd) and I'm using the following configuration: https://github.com/warm-metal/container-image-csi-driver/blob/v1.1.0/sample/ephemeral-volume.yaml It works for me as soon as I add the readOnly flag to the volume definition. |
|
@woehrl01 could you please share the logs from node-plugin daemonset pod which is running on the same node where the workload was deployed? |
|
|
Currently in our automated builds we are only testing the driver against k8s version v1.25. Here's the compatibility matrix info. I tested the driver on a $ kubectl logs ephemeral-volume-thb2m
+ '[' /target '!=' ]
+ '[' -f /target/csi-file1 ]
+ '[' -f /target/csi-file2 ]
+ '[' -d /target/csi-folder1 ]
+ '[' -f /target/csi-folder1/file ]
+ exit 0 |
|
@mugdha-adhav yes, your right the logs don't shoe any problems, that's why I asked if you have additional ideas to debug this. The error I recieve is a kubernetes event on the pod and it's created by kubelet: { "event.firstTimestamp": "2024-02-14T19:03:22Z", "event.involvedObject.apiVersion": "v1", "event.involvedObject.fieldPath": "spec.containers{ng}", "event.involvedObject.kind": "Pod", "event.involvedObject.name": "h-8648c99fbf-wkgrg", "event.involvedObject.namespace": "j", "event.involvedObject.resourceVersion": "1600765086", "event.involvedObject.uid": "7b39946a-5be40ee06e49b7d", "event.lastTimestamp": "2024-02-14T19:03:34Z", "event.message": "(combined from similar events): Error: failed to generate container \"218b131a8748748b7ba121c4a2fd5a6b182659fcecdff0357bd106aa1b1fcfb4\" spec: failed to apply OCI options: relabel \"/var/lib/kubelet/pods/7b39946a-5be4-49c4-8f52-20ee06e49b7d/volumes/kubernetes.io~csi/source/mount\" with \"system_u:object_r:data_t:s0:c211,c621\" failed: lsetxattr /var/lib/kubelet/pods/7b39946a-5be4-49c4-8f52-20ee06e49b7d/volumes/kubernetes.io~csi/source/mount/var: read-only file system", "event.metadata.creationTimestamp": "2024-02-14T19:03:22Z", "event.metadata.managedFields[0].apiVersion": "v1", "event.metadata.managedFields[0].fieldsType": "FieldsV1", "event.metadata.managedFields[0].manager": "kubelet", "event.metadata.managedFields[0].operation": "Update", "event.metadata.managedFields[0].time": "2024-02-14T19:03:34Z", "event.metadata.name": "h8648c99fbf-wkgrg.17b3d004cd0490fd", "event.metadata.namespace": "p67747", "event.metadata.resourceVersion": "1600786275", "event.metadata.uid": "ac9784a8-f562-48d4-b9be-fe926e9a3c13", "event.reason": "Failed", "event.source.component": "kubelet", "event.source.host": "ip--.f.compute.internal", "event.type": "Warning", "integrationName": "kube_events", "integrationVersion": "2.8.1", "old_event.count": 1, "old_event.firstTimestamp": "2024-02-14T19:03:22Z", "old_event.involvedObject.apiVersion": "v1", "old_event.involvedObject.fieldPath": "spec.containers{ng}", "old_event.involvedObject.kind": "Pod", "old_event.involvedObject.name": "h-8648c99fbf-wkgrg", "old_event.involvedObject.namespace": "h", "old_event.involvedObject.resourceVersion": "1600765086", "old_event.involvedObject.uid": "7b39946a-5beee06e49b7d", "old_event.lastTimestamp": "2024-02-14T19:03:22Z", "old_event.message": "(combined from similar events): Error: failed to generate container \"be6c7b7e8c3d4a3386a047312d173f6a94490d461e3073d6205cc1cf888f8f24\" spec: failed to apply OCI options: relabel \"/var/lib/kubelet/pods/7b39946a-5be4-49c4-8f52-20ee06e49b7d/volumes/kubernetes.io~csi/source/mount\" with \"system_u:object_r:data_t:s0:c211,c621\" failed: lsetxattr /var/lib/kubelet/pods/7b39946a-5be4-49c4-8f52-20ee06e49b7d/volumes/kubernetes.io~csi/source/mount/var: read-only file system", "old_event.metadata.creationTimestamp": "2024-02-14T19:03:22Z", "old_event.metadata.managedFields[0].apiVersion": "v1", "old_event.metadata.managedFields[0].fieldsType": "FieldsV1", "old_event.metadata.managedFields[0].manager": "kubelet", "old_event.metadata.managedFields[0].operation": "Update", "old_event.metadata.managedFields[0].time": "2024-02-14T19:03:22Z", "old_event.metadata.name": "h-8648c99fbf-wkgrg.17b3d004cd0490fd", "old_event.metadata.namespace": "h", "old_event.metadata.resourceVersion": "1600782603", "old_event.metadata.uid": "ac9784a8-f562-48d4-b9b", "old_event.reason": "Failed", "old_event.source.component": "kubelet", "old_event.source.host": ".compute.internal", "old_event.type": "Warning", "summary": "(combined from similar events): Error: failed to generate container \"218b131a8748748b7ba121c4a2fd5a6b182659fcecdff0357bd106aa1b1fcfb4\" spec: failed to apply OCI options: relabel \"/var/lib/kubelet/pods/7b39946a-5be4-49c4-8f52-20ee06e49b7d/volumes/kubernetes.io~csi/source/mount\" with \"system_u:object_r:data_t:s0:c211,c621\" failed: lsetxattr /var/lib/kubelet/pods/7b39946a-5be4-49c4-8f52-20ee06e49b7d/volumes/kubernetes.io~csi/source/mount/var: read-only file system", "timestamp": 1707937414000, "verb": "UPDATE" |
|
I just found the following related issue of a different csi driver with bottlerocket, looks like the issue is selinux + hostpath mount related: bottlerocket-os/bottlerocket#2556 A fix is described here by passing different mount labels: |
|
Interesting, it seems the issue is platform specific. We could add |
|
@mugdha-adhav I think it makes sense to add those different mount options to the helm chart. I also think that the context needs to be passed in the sourcecode. If I see it correct then the additonal mount options need to be set here, right before the container-image-csi-driver/pkg/backend/containerd/containerd.go Lines 38 to 45 in 3d36010 |
|
@woehrl01 would appreciate your help with sending out a PR for this. Please let me know if you want me to assign this issue to you. |
|
Hi, We had the same issue with trying to mount a Loki PVC on bottlerocket 1.29 Loki sets its data chunks to read/only after it has finished writing to them, this is what I assume is actually causing the failure (having files in the r/w mount being read only, so SELinux cannot relabel them).
|
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. |
|
@woehrl01 (or anyone else) would you be interested in sending a fix for this? I haven't worked with bottlerocket yet, so it might take me a bit longer to get the fix out. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
Would like to test out this really promissing csi driver, but I receive the following error:
I already changed the mount and the pod to be readable, but I still have that error.
I'm using EKS 1.28 with bottlerocket nodes.
Any ideas what I could try?
Edit: I got it working by setting
readOnly: trueon the volume directly. Any idea how I can troubleshoot why a writable volume does not work?Thanks!
The text was updated successfully, but these errors were encountered: