[CephFS] Permission denied for mounted directory except last created pod #3562
Comments
|
@lwj5 |
Rakshith-R
added
question
|
This is the deployment used for the test. Let me know what you would like changed apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
...
name: ceph
namespace: test
spec:
replicas: 2
...
template:
spec:
affinity: {}
containers:
- command:
- sleep
- "10000"
image: debian:bullseye-slim
imagePullPolicy: IfNotPresent
name: container
securityContext:
allowPrivilegeEscalation: false
capabilities: {}
privileged: false
readOnlyRootFilesystem: false
runAsUser: 1001
volumeMounts:
- mountPath: /ceph
name: vol-igx3b
...
securityContext:
fsGroup: 1001
volumes:
- name: vol-igx3b
persistentVolumeClaim:
claimName: cephfs |
|
@humblec |
|
We're seeing identical symptoms to the description above. For our case, it looks like SELinux category changes are resposible for the permission denial. A ReadWriteMany cephfs volume on the first pod looks fine: After a second pod mounts the volume, the first pod loses access: After ignoring the dontaudit rules ( After running I suspect when the second pod starts up, it causes a relabeling and changes the SELinux categories on the contents, resulting in a denial for the first pod. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions. |
|
Not stale |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions. |
|
still present |
|
@lwj5 this will be a problem with the SELinux relabelling on the cephfs mounts. When you create the second pod, the cephfs mountpoint will get relabelled with the second pod SELinux labels, and the first pod will start getting permission denied error. You must ensure that all the application pods using the same cephfs PVC should use the same SELinux labels. |
|
Thanks @Madhu-1 for your solution and @jthiltges for the diagnostic, this means that a static SELinux level must be set for every deployment. I wish there could be an better way. But nonetheless, thanks for the input. |
|
Thank you all, and I appreciate the info, though this is disappointing news. Having to set pods to the same category weakens the security benefits of SELinux. It would be less surprising if a |
Describe the bug
When creating a deployment with a ReadWriteMany cephFS PVC, only the last created pod has access to the mounted folder. The earlier pods will show Permission denied when
cdto that directory.If deployment is set to privilege, this does not happen.
I've looked at this #1097 but there is no denial in SELinux and I've set container_use_cephfs=1 as well to test.
Environment details
fuseorkernel. for rbd itskrbdorrbd-nbd) : kernelSteps to reproduce
Steps to reproduce the behavior:
Actual results
Permission denied on any earlier pods.
Expected behavior
All pods can access the volume
Logs
If the issue is in PVC mounting please attach complete logs of below containers.
plugin pod from the node where the mount is failing.
csi-cephfsplugin logs for node of 1st replica
csi-cephfsplugin logs for node of 2nd replica
Additional context
PVC in question is
pvc-74206ef3-4bcd-40e6-af26-40ec2b994a23ID of first pod
c3f729fe-2c56-41bf-aa53-b70e5ee17f10ID of second pod
68b1501c-e845-4f21-9abd-fb587d1da658See #1097
The text was updated successfully, but these errors were encountered: