SELinux prevents deployment of ceph-mgr after host package updates, EL8.7

[Venator-Fox]

Bug Report

What happened: After performing a dnf update -y to RHEL 8.7 and associated packages; ceph no longer successfully deploys due to an avc denial with keyrings for ceph-mgr. I don't think? this is an 8.7 issue but rather some other system packages that were updated which causes this. Setting selinux to permissive allows a deploy. An exact deployment on a host on 8.6 (with enforcing) last fully patched ~1 month ago is successful; updating to 8.7+ a month worth of updates to all other packages appears to cause this issue. No changes in the working repository from the deployment admin node.

What you expected to happen: Successful deployment with SELinux enforcing

How to reproduce it (minimal and precise):
Attempt to run a deployment on fully updated RHEL 8.7 and associated packages

The deployment will fail. Investigating a failing node it was noted the ceph manager container was not running. Attempting to start ceph-mgr@service will cause the podman container to immediately crash due to permission denial on keyrings. A restorecon on /etc/ceph and /var/lib/ceph with systemctl restart ceph-mgr@service did not seem to resolve the issue. The following denials are present in audit. Setting to permissive allows a deploy and will generate these messages:

type=AVC msg=audit(1673733722.343:7745): avc:  denied  { read } for  pid=55613 comm="ceph-mgr" name="keyring" dev="dm-0" ino=68051744 scontext=system_u:system_r:container_t:s0:c373,c524 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=AVC msg=audit(1673733722.343:7745): avc:  denied  { open } for  pid=55613 comm="ceph-mgr" path="/var/lib/ceph/mgr/ceph-lab-csp-sc1-301/keyring" dev="dm-0" ino=68051744 scontext=system_u:system_r:container_t:s0:c373,c524 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1

When enforcing, this causes the container to exit on startup

-- Unit ceph-mgr@cs-csp-io2-5a01.service has finished starting up.
-- 
-- The start-up result is done.
Jan 14 02:44:11 cs-csp-io2-5a01 ceph-mgr-csp-io2-5a01[50295]: find: '/var/lib/ceph/mgr/ceph-csp-io2-5a01/keyring': Permission denied
Jan 14 02:44:11 cs-csp-io2-5a01 ceph-mgr-csp-io2-5a01[50295]: chown: cannot access '/var/lib/ceph/mgr/ceph-csp-io2-5a01/keyring': Permission denied

Raw Audit Messages
type=AVC msg=audit(1673733722.343:7745): avc:  denied  { read } for  pid=55613 comm="ceph-mgr" name="keyring" dev="dm-0" ino=68051744 scontext=system_u:system_r:container_t:s0:c373,c524 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1


type=AVC msg=audit(1673733722.343:7745): avc:  denied  { open } for  pid=55613 comm="ceph-mgr" path="/var/lib/ceph/mgr/ceph-lab-csp-sc1-301/keyring" dev="dm-0" ino=68051744 scontext=system_u:system_r:container_t:s0:c373,c524 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1


type=SYSCALL msg=audit(1673733722.343:7745): arch=x86_64 syscall=openat success=yes exit=EIO a0=ffffff9c a1=5573c93aaf90 a2=80000 a3=0 items=0 ppid=55553 pid=55613 auid=4294967295 uid=167 gid=167 euid=167 suid=167 fsuid=167 egid=167 sgid=167 fsgid=167 tty=(none) ses=4294967295 comm=ceph-mgr exe=/usr/bin/ceph-mgr subj=system_u:system_r:container_t:s0:c373,c524 key=(null)ARCH=x86_64 SYSCALL=openat AUID=unset UID=unknown(167) GID=unknown(167) EUID=unknown(167) SUID=unknown(167) FSUID=unknown(167) EGID=unknown(167) SGID=unknown(167) FSGID=unknown(167)

Hash: ceph-mgr,container_t,var_lib_t,file,read

This will in turn cause deployment to fail when it expects ceph-mgr

TASK [ceph-mgr : wait for all mgr to be up] *******************************************************************************************************************************************************************                                                                 
Saturday 14 January 2023  14:52:11 -0700 (0:00:00.131)       0:04:07.554 ******                                                                                                                                                                                                 
FAILED - RETRYING: wait for all mgr to be up (30 retries left).       
FAILED - RETRYING: wait for all mgr to be up (29 retries left).            
<...>
FAILED - RETRYING: wait for all mgr to be up (1 retries left).
fatal: [dev-csp-sc1-303 -> dev-csp-sc1-301]: FAILED! => changed=false 
  attempts: 30

Setting all nodes to permissive or adding a catchall local policy module appears to allow deployment but both cases are less than ideal.

Environment:

• OS (e.g. from /etc/os-release): PRETTY_NAME="Red Hat Enterprise Linux 8.7 (Ootpa)"
• Kernel (e.g. uname -a): 4.18.0-425.10.1.el8_7.x86_64 SMP Wed Dec 14 16:00:01 EST 2022 x86_64 x86_64 x86_64 GNU/Linux
• Docker version if applicable (e.g. docker version): podman 4.2.0
• Ansible version (e.g. ansible-playbook --version): ansible-playbook 2.10.17
• ceph-ansible version (e.g. git head or tag or stable branch): tag v6.0.28 d7bf53a
• Ceph version (e.g. ceph -v): 16.2.10 via quay.io/ceph/daemon:v6.0.9-stable-6.0-pacific-centos-stream8-x86_64

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.