SELinux blocking disk operations on a hostPath volume

[mcg1969]

I have a hostPath persistent volume pointed to a path /opt/anaconda/storage that is owned by 1000:0, the same UID and GID that the pod will run in. This PV is shared by three pods, one of them running postgres. This pod tries to chmod 700 its subdirectory (because, you know, postgres). I believe SELinux is denying that operation, at least sometimes. The symptom is that when I first start the pod, it works fine, and it can see /mnt/pgdata (where /opt/anaconda/storage/pgdata is mounted), but after some time, it can no longer access that directory, despite having the right owner and primary group.

Disabling SELinux solved the problem, but of course I'd prefer not to have to do that. This seems like it may be a duplicate of #100 but I'm not sufficiently familiar with SELinux yet to know how to test the chcon fix. I will update if I find it.

I turned on permissive SELinux mode and fired up the application, and got a much longer set of denials. I'm offering that in the attachment below. Here's what seems to be the relevant snippet of the log before I turned permissive mode on.

type=PROCTITLE msg=audit(11/05/2023 20:04:14.350:980) : proctitle=chmod 00700 /mnt/pgdata 
type=SYSCALL msg=audit(11/05/2023 20:04:14.350:980) : arch=x86_64 syscall=fchmodat success=no exit=EACCES(Permission denied) a0=AT_FDCWD a1=0x56175e3f04d0 a2=0700 a3=0x7f1174880f98 items=0 ppid=109618 pid=109647 auid=unset uid=ec2-user gid=root euid=ec2-user suid=ec2-user fsuid=ec2-user egid=root sgid=root fsgid=root tty=(none) ses=unset comm=chmod exe=/usr/bin/chmod subj=system_u:system_r:container_t:s0:c173,c969 key=(null) 
type=AVC msg=audit(11/05/2023 20:04:14.350:980) : avc:  denied  { setattr } for  pid=109647 comm=chmod name=pgdata dev="nvme0n1p2" ino=8923402 scontext=system_u:system_r:container_t:s0:c173,c969 tcontext=system_u:object_r:container_file_t:s0:c45,c627 tclass=dir permissive=0 

selinux.log

[mcg1969]

This did not, unfortunately, solve the problem:

sudo chcon -R -t svirt_sandbox_file_t -l s0 /opt/anaconda/storage

[mcg1969]

I was able to get the full application running this command repeatedly as the containers started up.

sudo chcon -R -t svirt_sandbox_file_t -l s0 /opt/anaconda/storage

Because three different pods are interacting with this directory, and Postgres is particularly feisty about ownership, the various operations they do to confirm ownership and set permissions causes the pods to lose access. Here's the audit log:

----
type=PROCTITLE msg=audit(11/06/2023 15:26:59.770:300) : proctitle=chmod 00700 /mnt/pgdata 
type=SYSCALL msg=audit(11/06/2023 15:26:59.770:300) : arch=x86_64 syscall=fchmodat success=no exit=EACCES(Permission denied) a0=AT_FDCWD a1=0x558dc11ee4d0 a2=0700 a3=0x7f075ab07f98 items=0 ppid=14261 pid=14317 auid=unset uid=ec2-user gid=root euid=ec2-user suid=ec2-user fsuid=ec2-user egid=root sgid=root fsgid=root tty=(none) ses=unset comm=chmod exe=/usr/bin/chmod subj=system_u:system_r:container_t:s0:c813,c975 key=(null) 
type=AVC msg=audit(11/06/2023 15:26:59.770:300) : avc:  denied  { setattr } for  pid=14317 comm=chmod name=pgdata dev="nvme0n1p2" ino=511705289 scontext=system_u:system_r:container_t:s0:c813,c975 tcontext=system_u:object_r:container_file_t:s0:c79,c834 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(11/06/2023 15:40:56.187:543) : proctitle=postgres: walwriter    
type=SYSCALL msg=audit(11/06/2023 15:40:56.187:543) : arch=x86_64 syscall=pwrite success=no exit=EACCES(Permission denied) a0=0x3 a1=0x7f86a2678000 a2=0x2000 a3=0x866000 items=0 ppid=36573 pid=36596 auid=unset uid=ec2-user gid=root euid=ec2-user suid=ec2-user fsuid=ec2-user egid=root sgid=root fsgid=root tty=(none) ses=unset comm=postgres exe=/usr/lib/postgresql/12/bin/postgres subj=system_u:system_r:container_t:s0:c250,c1019 key=(null) 
type=AVC msg=audit(11/06/2023 15:40:56.187:543) : avc:  denied  { write } for  pid=36596 comm=postgres path=/mnt/pgdata/pg_wal/000000010000000000000001 dev="nvme0n1p2" ino=58720526 scontext=system_u:system_r:container_t:s0:c250,c1019 tcontext=system_u:object_r:container_file_t:s0:c428,c438 tclass=file permissive=0

[mcg1969]

OK, here's what worked for me.

1. Run sudo chcon -R -t svirt_sandbox_file_t -l s0 /opt/anaconda/storage
2. Launch one of the deployments in isolation; let it stabilize and interact with the volume.
3. Examine ls -Zd /opt/anaconda/storage to obtain the auto-assigned MCS label for the container: s0:c248,c366, for isntance
4. Add this label to the securityContext for the deployments, including the one just launched.

   securityContext:
       runAsUser: 1000
       seLinuxOptions:
         level: s0:c248,c366
   

5. Bring up the rest of the deployments.

[mcg1969]

Sorry, it's me again. I've been comparing k3s and rke2 here, which ostensibly use the same local path provisioner, and k3s is not having the same issue. And it seems to be because k3s is not applying the same ownership semantics on the volume. In particular, here's what ls -lZd gives me for the Postgres volume:

drwx------. 19 ec2-user root system_u:object_r:usr_t:s0 4096 Nov  8 14:23 pgdata

Note the lack of container labels.

I'm just going to check myself that perhaps k3s doesn't enable full SELinux enforcement by default; I will see. If that's the case, no need to reply, I'll figure that out soon enough :-)

EDIT: installing k3s with --selinux produces the same behavior, so it's definitely generic

[jcox10]

We create a selinux policy to make it work. I don't remember how we figured this out, but we have our ansible do it automatically now.

1. Create /root/localpathpoicy.te

   module localpathpolicy 1.0;
   
   require {
       type usr_t;
       type init_t;
       type container_t;
       type container_var_lib_t;
       class dir { search write add_name create remove_name rmdir setattr getattr };
       class file { create open write append read unlink setattr getattr };
   }
   
   #============= container_t ==============
   allow container_t container_var_lib_t:file { create open write append read setattr getattr unlink };
   allow container_t container_var_lib_t:dir { add_name create remove_name rmdir setattr write search };
   allow container_t init_t:dir search;
   allow container_t usr_t:dir { add_name create remove_name rmdir setattr getattr write };
   allow container_t usr_t:file { create unlink write setattr getattr };
   allow container_t init_t:file { read open };
   

2. checkmodule -M -m -o /root/localpathpolicy.mod /root/localpathpolicy.te
3. semodule_package -o /root/localpathpolicy.pp -m /root/localpathpolicy.mod
4. semodule -i /root/localpathpolicy.pp
5. semanage fcontext -a -t container_file_t "/var/lib/localpath(/.*)?" (replace /var/lib/localpath with your path on disk)
6. restorecon -R -v /var/lib/localpath

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.