SELinux prevents access to existing sub-directories in statically provisioned PVs #481
Comments
gfschmidt
added
the
Type: Bug
|
It seems that access to the sub-directories of a statically provisioned PV is not prevented by SELinux if the static PV itself is not backed by the root fileset of an IBM Spectrum Scale file system directly (e.g. mounted at /mnt/fs1). If the static PV is instead backed by a sub-directory located under the IBM Spectrum Scale file system's root directory (e.g. /mnt/fs1/nfs-export) then access to sub-directories in that directory works as expected and is not prevented by SELinux. For example, if we only want to grant access to a sub-directory /mnt/fs1/nfs-export in IBM Spectrum Scale and not to the entire IBM Spectrum Scale file system mounted at /mnt/fs1 and all its sub-directories then it seems to work ok. With a static PV backed by a sub-directory /mnt/fs1/nfs-export in IBM Spectrum Scale and a PVC mounted in the pod at /data/fs1/nfs-export we have full access to all sub-directories and their files of this backing directory: It seems that SELinux prevents access to sub-directories of the backing directory in a statically provisioned PV only in case the static PV's volumeHandle (e.g. "17399599334533218545;099B6A7A:5EB99721;path=/mnt/fs1") is pointing directly to the root fileset of an IBM Spectrum Scale file system (i.e. the root of an entire IBM Spectrum Scale file system). In this case when granting access to an entire IBM Spectrum Scale file system with a statically provisioned PV (e.g. path=/mnt/fs1) the expected behavior would be to have full access to all sub-directories as determined by the individual file system permissions and access not to be prevented by SELinux on the worker nodes. |
|
Looks like in case of root fileset PVC, OCP did not relabel the existing subdirectory. Saying because of following selinux log-
Can you verify by running ?
Few other things to check-
|
|
Storage cluster SELinux settings for this Issue are: [root@fscc-sr650-12 ~]# mmdsh -Nall -f1 getenforce When I first run into this issue I had SELinux : Disabled on all nodes in the storage cluster which was supposed to be the default on the ESS. When I opened this issue I already had changed the setting to Enforcing and also freshly re-created the directory nfs-export which was then used for the exercise described in this issue. If the SELinux settings on the storage cluster do matter then what is the required SELinux setting on the IBM Spectrum Scale storage cluster / ESS used with CNSA/CSI on OpenShift? |
|
File listing with ls -lZ on storage and client cluster Note that most of the other directories here were created when SELinux was disabled. The nfs-export directory referred to in this Issue was freshly created after enabling SELinux: Enforcing on the storage cluster. The output from the CNSA cluster and storage cluster looks the same to me. All files in the nfs-export sub-directory carry the same SELinux flags: Storage cluster: CNSA: |
|
Please also note my first comment.
So it only fails when mounting an entire Spectrum Scale root file system via a static PV. In both cases the pod is directly executed by an OpenShift admin user with cluster role cluster-admin. The pod is even running privileged for the time being. So I assume the SCC that will be used is the privileged SCC which grants unlimited permissions with |
|
@amdabhad Please check this. |
|
Issue 1. When a directory A is used for a PV, and in turn used by a pod then inside the pod, the sub-directories of A are not accessible.
Issue 2. When multiple pods are accessing same/nested path - the pods created earlier lose access to the common path due to relabeling.
b. In order for multiple pods to have access to a common path, the pods must use above SCC. @gfschmidt Please have a look at these solutions and let me know if any concerns. |
|
@amdabhad, thanks. I will be running some tests and will share my results soon. |
|
The instructions provided above are workarounds as proposed similarly in Considerations for mounting read-write many (RWX) volumes and may help to fix given conditions when running into them. None of these manual workarounds should typically be required when provisioning static PVs. The point of this issue is that the SELinux relabeling is applied differently when using static provisioning:
So there is a difference in how a static PV is handled with regard to the mount point in the volumeHandle. No SELinux relabeling is happening with static PVs if the mount point of an entire IBM Spectrum Scale file system is given in the volumeHandle. Even if SELinux relabeling is not wanted in some cases the expected behavior is that it should happen in the same way as with a regular directory to ensure that a pod has access to the data in the PV. The creation of the static PV is in the domain of an admin (not the user). So the creation is a manual process similar to all static PVs and intentional. IBM Spectrum Scale supports multiple file systems and there may be cases where an entire Spectrum Scale file system (similar to a directory in IBM Spectrum Scale) is shared and made available to OpenShift users through static provisioning (i.e. a static PV). With proper SELinux relabeling and the proposed methods described in point 3 and 4 below parallel access to the data can be established with the proper SELinux context in the same way as with regular directories. I observed the following behavior in OpenShift (here with /mnt/fs1 = root of IBM Spectrum Scale file system): Expected behavior (access granted on directory level beneath the IBM Spectrum Scale root mount point):
In this case the SELinux context can be controlled on pod level by the user if permitted by the SCC under which the user is running. The SCC is controlled by the admin. Method 3 (per namespace) and method 4 (per pod/container) allow to share parallel access to the same data directory in IBM Spectrum Scale through static provisioning. These two approaches are simple to apply. Defining a specific SELinux context in the SCC as proposed above requires additional efforts (configuration of roles, rolebindings, service accounts). -> Especially method 3 based on the namespace annotation may be worth being added to the CSI documentation, e.g. in Considerations for mounting read-write many (RWX) volumes. Unexpected behavior (access denied when using an IBM Spectrum Scale root mount point):
-> Manual adjustment of the SELinux labels on the storage cluster as proposed above should not be the regular solution here - it's rather a workaround to fix a given situation. Manual adjustment of the SELinux labels on the storage cluster is also not needed when working on a regular directory (one level under the root mount point of the IBM Spectrum Scale file system) - here everything works as expected. -> Furthermore, a definitive direction if SELinux needs to be enabled or disabled on the IBM Spectrum Scale storage cluster (i.e. set to enforcing/permissive/disabled and targeted /minimum/mls) is missing in the official documentation. As far as I know the default setting with IBM ESS SELinux is typically disabled on the io nodes. Nested directories
|
|
Summary of open issues:
|
|
@gfschmidt , thank you for the detailed try outs and updates in this issue. HPO team is also at the same state that the IBM SS FS (/mnt/fs1) is mounted onto the Noobaa pod using the namespacestore which will vary whenever we reinstall Noobaa on the OCP cluster. We are in 5121 stream and there is no proper automatic method to set this looks like On SC we are doing Permissive and we have issues when working on the complete solution of creating objects etc -- We mount the /mnt/fs1 as is on the pod (static PV) @amdabhad / CSI team this has been discussed with @deeghuge few times with live debugging and we need an E2E solution without any manual changes on SC. Noobaa suggested a solution to openup c0.c1023 (as temporary workaround) until this issue is fixed, which we see as a security loop hole by HPO team. Can we reanalyze and get the Fix here ? |
|
@gfschmidt --> you can refer to Noobaa defect 6761 that has all the details w/r/t scc changing for each install of Noobaa in openshift-storage ns |
|
@amdabhad Cluster rebooted for us as shown below mmdsh -N all uptime |
|
@rkomandu If you are following step 1, I think you will have to perform this step again when the nodes are rebooted, because I have seen the scale operations like |
|
When mounting the root of the volume - how does the Pod definition looks like and how does the mount appear on the node where pod was scheduled? i.e - can you post precise mount options that was visible once volume was mounted on the node? |
|
Hi @gnufied , On worker node filesystem is mounted at /var/mnt/gpfs1. This is the describe output of pv,pvc and pod /mnt/gpfs1 and /var/mnt/gpfs1 is same thing on worker node Before pod actually see data, there is additional softlink get created on host which looks like as this |
|
The spec says: And in this case target path is just a symlink. The first problem is - since target path is symlink, selinux detection does not work. We expect target path to be a mount point. |
|
@gnufied it does work with softlink when softlink point to directory inside the filesystem for example if we use |
|
how does the target path looks like when softlink points to a directory inside filesystem? The bottom line is - SELinux detection currrently is not working in k8s if target path is a symlink. If you think it works correctly if volume is a subdirectory - then I would say it may be completely a fluke(I would not trust the result). Also, who is creating mount points like |
|
This is for directory inside filesystem Once pod started, the softlink on worker node looks like And the SELinux labels after pod running looks like
|
|
Once pod with subdirectory is running, can you post the output of |
|
|
This is rather surprising. The code in kubernetes works like following:
We do not evaluate the symlink before determing the selinux support and hence it should not work. We can setup a call to go over this if you prefer. You can find me on kubernetes slack or coreos (redhat) slack. |
|
@deeghuge why the CSI driver uses symlinks instead of mounts? Esp. when bind-mounts are cheap? |
|
@jsafrane Here is some background on why softlink was chosen over bind-mounts |
|
If I understand it correctly, the CSI driver has only That's... unexpected.
This requires someone to configure and mount GPFS on every single host. That's defeats the purpose of CSI - details about the storage should be in the driver container and not on the host. I know the line between the host and CSI driver can be blurry, e.g. iscsid and multipathd can run on the host, still, having full storage mounted on the host is something I've never seen. Did you consider mounting the GPFS volumes inside the driver container, e.g. based on a ConfigMap with all the filesystems to mount, and then providing bind-mounts from it? |
|
I agree with your suggestions but there is some background/design aspect for the approach taken
|
|
Just for further reference: In addition to my comment with the SELinux namespace annotations above (see gfschmidt commented on Oct 26, 2021 I summarized major considerations for using static provisioning with shared data access in IBM Spectrum Scale on OpenShift in the following blog post: Advanced Static Volume Provisioning with IBM Spectrum Scale on Red Hat OpenShift (i.e. distinct mapping of static PVs to PVCs using labels or claimRef, ensure proper uid/gid and SELinux context with namespace annotations or pod/container securityContext ). |
|
@deeghuge could you please help put FQI labels? |
deeghuge
added
Customer Probability: Medium
|
Just some additional information on this SELinux relabeling topic that might be helpful for anyone interested: If someone wants to see SELinux relabeling in action and understand how the standard SELinux relabling works when pods in different namespaces access the same data on IBM Spectrum Scale I recorded and shared some demos: The first demo shows what happens when the same data in IBM Spectrum Scale is accessed by three non-privileged users in OpenShift from three namespaces without taking the SELinux security context into account. The users are regular users running in their own namespaces under the "restricted" SCC (security context constraints). Here the "restricted" SCC enforces a MustRunAs policy on the SELinux security context and the default value for the SELinux MCS label is taken from the pre-allocated values given in the annotations of the namespace where the pod is running. So the pod in each namespace runs with another default SELinux MCS label and as soon as a pod is started and mounts the volume the whole data in the mounted volume is relabeled with the default SELinux MCS label of the namespace. In this case each user shuts out all other users from accessing the pods if no further precautions with regard to SELinux relabeling are taken. This second demo shows how you can define and apply a custom SCC to ensure that different non-privileged users in three namespaces can safely access the same data in IBM Spectrum Scale across worker nodes in a RWX (read-write-many) access mode with a proper SELinux context and individual uid/gid file permissions. We show how to grant access to a custom SCC ("shared-scc") via a service account (plus role and rolebinding) in the user namespace as well as by assigning the custom SCC directly to the user (could also be a group of users). Here the pods in each namespace run with the same SELinux MCS label as defined in the custom SCC ("shared-scc"). The PVs have been provisioned as shown in the first demo above. The PVs for the two demos have been provisioned as shown in the this demo: ADVANCED VOLUME PROVISIONING FOR DATA SHARING. This presentation gives a good summary of the SELinux issue and how to safely mitigate it with custom SCCs. The setup used in the videos and more information is available in my blog post here. SELinux relabeling may also lead to ContainerCreationErrors on large volumes with many files under some circumstances if a timeout of 120sec is hit and the relabeling did not successfully finish in that time. My blog post above also shows one way how to disable SELinux relabeling on a volume today by using a custom SCC and "spc_t" in this section which is based on one of two solutions proposed in a Red Hat artice. |
|
Note that a similar behavior as described in this issue may also be observed if the backing directory of a statically provisioned volume is actually an IBM Spectrum Scale independent fileset (instead of a regular directory). An independent fileset may behave in a similar way when specified as backing directory in the volumeHandle of a statically provisioned volume . In this case automatic SELinux relabeling is eventually skipped by OpenShift on any sub-directories. This can be seen as an advantage for static provisioning and shared data access as the pre-existing permissions and SELinux attributes in the file system / fileset are honored. Here the storage admin keeps control over the access permissions of the data as set in the file system and can generally or selectively grant access to applications in OpenShift. For example, the storage admin could grant general access to the data in an independent fileset accessed by pods/containers in OpenShift through a statically provisioned PV by setting the SELinux MCS label on an entire fileset (or just selected sub-directories or objects within the fileset) to "system_u:object_r:container_file_t:s0" on the IBM Spectrum Scale storage cluster (Note: The option -R will recursively change the SELinux label on all files and directories at the specified destination): This would only have to be done once as the applied SELinux settings will persist. If SELinux is not enabled on the storage cluster the manual SElinux relabeling could also be done by a cluster admin on OpenShift by running a debug pod, for example Note: As always when executing commands as root user: Be very cautious and careful when doing the SELinux relabeling by running the chcon command directly in a debug pod on an OpenShift worker node. When done wrong you can harm your system! Without adding a specific category (like c7, c28) to the SELinux MCS label "container_file_t:s0" the data can be accessed by any pod/container independent of the specific SELinux MCS category that is assigned to the pod/container process by OpenShift like, for example, "container_t:s0:c7,c28" because the empty set of SELinux categories is also part of every process set. Any pod/container means, of course, any pod/container that has been given explicit access to the data through a statically provisioned PV. The storage admin can even set the SELinux MCS labels more selectively with specific categories on selected (or all) objects in the independent fileset if needed but that would also entail to carefully align the SELinux labels in the file system with the SELinux security context of the pods/containers in OpenShift which is dependent on the OpenShift Security Context Constraints (SCCs) as well as the user, group, service account and pre-allocated defaults from the namespace. Please refer to my blog post here for more information. |
|
Issue if fixed in CNSA 5.1.7. Please reopen if you still see issue on latest CNSA. |
Describe the bug
When using static provisioning in OpenShift 4.7 with Scale CSI v2.2.0 to gain access to an entire IBM Spectrum Scale file system (e.g. mounted at /mnt/fs1 in CNSA) with existing data then SELinux on the worker node prevents access to existing sub-directories even when running the pod under OpenShift with cluster-admin role. This was observed with IBM Spectrum Scale CNSA v5.1.1.1 and CSI v2.2.0 on x86 and OpenShift 4.7.17.
To Reproduce
File system attributes on CNSA:
File system attributes on remote storage cluster:
We create a static PV for the directory /mnt/fs1 on CNSA which is backed by a remote filesystem /gpfs/ess3000_1M on a remote storage cluster.
In order to allow a direct binding between the PVC and the specific PV holding the specific data we make use of Kubernetes labels and use a faked storage class name "static" which is used as some sort of annotation or identifier rather than a real storage class (which it isn't) - this prevents that the default storage class (if any should be defined) is applied instead rather than matching the specific PV from the pool of statically provisioned PVs with the PVC and its specific labels (in case the storage class is omitted in the PVC).
The following PVC achieves a direct 1:1 match between the claim and the specifc PV created above with the labels and the faked storage class name as selectors:
It appears we have full RW access to the root or first level of the file system backed by the static PV (/mnt/fs1) but we do not have any access to existing sub-directories in that directory /mnt/fs1/nfs-export). Even if the sub-directory's permissions is set to chmod 777.
Here, the statically provisioned PV is backed by the directory /mnt/fs1 in IBM Spectrum Scale which holds the sub-directory "nfs-export".
Expected behavior
When using static provisioning with IBM Spectrum Scale CSI to grant access to an existing directory (i.e. existing data) in IBM Spectrum Scale then it is expected that access to the sub-directories in that directory is available and not prevented by SELinux (as long as the file system UID/GID permissions allow it).
Environment
The text was updated successfully, but these errors were encountered: