Fresh install on STIG'd RockyLinux 9 fails to launch - etcd permission denied

[dajester2013]

Environmental Info:
RKE2 Version:
rke2 version v1.25.10+rke2r1 (e0c376c)
go version go1.19.9 X:boringcrypto

Node(s) CPU architecture, OS, and Version:
Linux ... 5.14.0-284.11.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Tue May 9 17:09:15 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
Single server

Describe the bug:
I am experiencing this on a freshly-installed RockyLinux 9 with DoD STIG profile applied at installation. Created the etcd user prior to launch as recommended in the documentation - uid/gid 982. Disabling fapolicy and selinux does not fix the permission denied error.

config.yaml:

selinux: true
profile: "cis-1.23"
write-kubeconfig-mode: "0600"
kube-controller-manager-arg:
  - "tls-min-version=VersionTLS12"
  - "tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
kube-scheduler-arg:
  - "tls-min-version=VersionTLS12"
  - "tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
kube-apiserver-arg:
  - "tls-min-version=VersionTLS12"
  - "tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
  - "enable-admission-plugins=ValidatingAdmissionWebhook,NodeRestriction,PodSecurityPolicy"
  - "request-timeout=300s"

# Enable only when auditing is enabled, blocks API when audit fails
#- "audit-log-mode=blocking-strict"

kubelet-arg:
  - "feature-gates=DynamicKubeletConfig=false"
  - "protect-kernel-defaults=true"
  - "streaming-connection-idle-timeout=5m" 

debug: true

crictl logs for etcd container:

{"level":"info","ts":"2023-06-02T18:15:03.754Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","--config-file=/var/lib/rancher/rke2/server/db/etcd/config"]}
{"level":"warn","ts":"2023-06-02T18:15:03.755Z","caller":"etcdmain/etcd.go:75","msg":"failed to verify flags","error":"open /var/lib/rancher/rke2/server/db/etcd/config: permission denied"}

Steps To Reproduce:

• Install RockyLinux9, with DoD STIG security profile applied at installation
• Installed RKE2
• Follow RHEL/Rocky guides to create etcd user, apply sysctl and networkmanager fixes.
• Use above config.yaml

Expected behavior:
RKE2 boots

Actual behavior:
etcd fails to run - permission denied to etcd config

Additional context / logs:

[brandond]

What are the permissions on the RKE2 DB directory? Has your hardening altered the permissions on /var/lib/rancher/rke2/server/db such that the etcd user does not have access to files within that directory?

[dajester2013]

[root@simplek8s etcd]# ls -alZ
total 8
drwx------. 2 etcd etcd unconfined_u:object_r:container_var_lib_t:s0   32 May 31 14:46 .
drwx------. 3 root root unconfined_u:object_r:container_var_lib_t:s0   18 May 31 14:46 ..
-rw-------. 1 etcd etcd unconfined_u:object_r:container_var_lib_t:s0 1119 Jun  2 14:30 config
-rw-------. 1 etcd etcd unconfined_u:object_r:container_var_lib_t:s0   18 May 31 14:46 name

[root@simplek8s etcd]# cat /etc/shadow | grep etcd
etcd:!!:19508::::::

I'm not sure what's causing the issue, I can run alpine in podman and list the contents:

[root@simplek8s /]# podman run -ti --rm -u 982 -v /var/lib/rancher/rke2/server/db/etcd:/etcd:Z alpine sh -c 'cat /etcd/config'
advertise-client-urls: https://10.112.35.155:2379
client-transport-security:
  cert-file: /var/lib/rancher/rke2/server/tls/etcd/server-client.crt
  client-cert-auth: true
  key-file: /var/lib/rancher/rke2/server/tls/etcd/server-client.key
  trusted-ca-file: /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
data-dir: /var/lib/rancher/rke2/server/db/etcd
election-timeout: 5000
experimental-initial-corrupt-check: true
heartbeat-interval: 500
initial-advertise-peer-urls: https://10.112.35.155:2380
initial-cluster: simplek8s-9553228c=https://10.112.35.155:2380
initial-cluster-state: new
listen-client-urls: https://127.0.0.1:2379,https://10.112.35.155:2379
listen-metrics-urls: http://127.0.0.1:2381
listen-peer-urls: https://127.0.0.1:2380,https://10.112.35.155:2380
log-outputs:
- stderr
logger: zap
name: simplek8s-9553228c
peer-transport-security:
  cert-file: /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.crt
  client-cert-auth: true
  key-file: /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.key
  trusted-ca-file: /var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt
snapshot-count: 10000

[brandond]

How did you install rke2; from tarball or from RPM? If you installed from RPM, did the rke2-selinux installation complete successfully?

[dajester2013]

I installed using the quick-start script - using RPM I assume. Everything installed correctly as far as I can tell.

I have found it will start if I do not enable the CIS-1.23 profile.

[dajester2013]

Also, if I attempt to run the podman container as non root, I get the same permission denied failure.

[jshaffer@simplek8s ~]$ podman run -ti --rm -u 982 -v /var/lib/rancher/rke2/server/db/etcd:/etcd:Z alpine sh -c 'cat /etcd/config'
Error: statfs /var/lib/rancher/rke2/server/db/etcd: permission denied

[brandond]

Can you check your audit log for denials? If possible you might also confirm that the rke2-selinux RPM install completed successfully, distros have recently updated their container-selinux package, and that has caused some conflicts that may have been missed in the rest of the install script output.

[dajester2013]

The only denials I see are from my attempts to use podman to test access:

[root@simplek8s etcd]#  ausearch -m avc
Email option is specified but /usr/lib/sendmail doesn't seem executable.
----
time->Wed May 31 15:01:31 2023
node=simplek8s type=PROCTITLE msg=audit(1685559691.252:12545748): proctitle=63617400636F6E666967
node=simplek8s type=PATH msg=audit(1685559691.252:12545748): item=0 name="config" inode=1074634824 dev=fd:07 mode=0100600 ouid=982 ogid=982 rdev=00:00 obj=unconfined_u:object_r:container_var_lib_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
node=simplek8s type=CWD msg=audit(1685559691.252:12545748): cwd="/etcd"
node=simplek8s type=SYSCALL msg=audit(1685559691.252:12545748): arch=c000003e syscall=2 success=no exit=-13 a0=7ffc22dccf4f a1=8000 a2=0 a3=0 items=1 ppid=850042 pid=850561 auid=1001 uid=982 gid=982 euid=982 suid=982 fsuid=982 egid=982 sgid=982 fsgid=982 tty=pts0 ses=11 comm="cat" exe="/bin/busybox" subj=system_u:system_r:container_t:s0:c715,c771 key="access"
node=simplek8s type=AVC msg=audit(1685559691.252:12545748): avc:  denied  { read } for  pid=850561 comm="cat" name="config" dev="dm-7" ino=1074634824 scontext=system_u:system_r:container_t:s0:c715,c771 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
----
time->Wed May 31 15:01:53 2023
node=simplek8s type=PROCTITLE msg=audit(1685559713.396:12546381): proctitle=63617400636F6E666967
node=simplek8s type=PATH msg=audit(1685559713.396:12546381): item=0 name="config" inode=1074634824 dev=fd:07 mode=0100600 ouid=982 ogid=982 rdev=00:00 obj=unconfined_u:object_r:container_var_lib_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
node=simplek8s type=CWD msg=audit(1685559713.396:12546381): cwd="/etcd"
node=simplek8s type=SYSCALL msg=audit(1685559713.396:12546381): arch=c000003e syscall=2 success=yes exit=3 a0=7fff385b7f4f a1=8000 a2=0 a3=0 items=1 ppid=850042 pid=851920 auid=1001 uid=982 gid=982 euid=982 suid=982 fsuid=982 egid=982 sgid=982 fsgid=982 tty=pts0 ses=11 comm="cat" exe="/bin/busybox" subj=system_u:system_r:container_t:s0:c715,c771 key=(null)
node=simplek8s type=AVC msg=audit(1685559713.396:12546381): avc:  denied  { open } for  pid=851920 comm="cat" path="/etcd/config" dev="dm-7" ino=1074634824 scontext=system_u:system_r:container_t:s0:c715,c771 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file permissive=1
node=simplek8s type=AVC msg=audit(1685559713.396:12546381): avc:  denied  { read } for  pid=851920 comm="cat" name="config" dev="dm-7" ino=1074634824 scontext=system_u:system_r:container_t:s0:c715,c771 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file permissive=1
----
time->Mon Jun  5 15:36:35 2023
node=simplek8s type=PROCTITLE msg=audit(1685993795.558:12569691): proctitle=636174002F657463642F6E616D65
node=simplek8s type=PATH msg=audit(1685993795.558:12569691): item=0 name="/etcd/name" inode=1074634823 dev=fd:07 mode=0100600 ouid=982 ogid=982 rdev=00:00 obj=unconfined_u:object_r:container_var_lib_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
node=simplek8s type=CWD msg=audit(1685993795.558:12569691): cwd="/"
node=simplek8s type=SYSCALL msg=audit(1685993795.558:12569691): arch=c000003e syscall=2 success=yes exit=3 a0=7fff6ee03f5b a1=8000 a2=0 a3=0 items=1 ppid=917610 pid=917624 auid=1001 uid=982 gid=0 euid=982 suid=982 fsuid=982 egid=0 sgid=0 fsgid=0 tty=pts0 ses=21 comm="cat" exe="/bin/busybox" subj=system_u:system_r:container_t:s0:c617,c841 key=(null)
node=simplek8s type=AVC msg=audit(1685993795.558:12569691): avc:  denied  { open } for  pid=917624 comm="cat" path="/etcd/name" dev="dm-7" ino=1074634823 scontext=system_u:system_r:container_t:s0:c617,c841 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file permissive=1
node=simplek8s type=AVC msg=audit(1685993795.558:12569691): avc:  denied  { read } for  pid=917624 comm="cat" name="name" dev="dm-7" ino=1074634823 scontext=system_u:system_r:container_t:s0:c617,c841 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file permissive=1
----
time->Mon Jun  5 15:37:14 2023
node=simplek8s type=PROCTITLE msg=audit(1685993834.340:12570034): proctitle=636174002F657463642F6E616D65
node=simplek8s type=PATH msg=audit(1685993834.340:12570034): item=0 name="/etcd/name" inode=1074634823 dev=fd:07 mode=0100600 ouid=982 ogid=982 rdev=00:00 obj=unconfined_u:object_r:container_var_lib_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
node=simplek8s type=CWD msg=audit(1685993834.340:12570034): cwd="/"
node=simplek8s type=SYSCALL msg=audit(1685993834.340:12570034): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe22bfbf5b a1=8000 a2=0 a3=0 items=1 ppid=917829 pid=917844 auid=1001 uid=982 gid=0 euid=982 suid=982 fsuid=982 egid=0 sgid=0 fsgid=0 tty=pts0 ses=21 comm="cat" exe="/bin/busybox" subj=system_u:system_r:container_t:s0:c289,c356 key="access"
node=simplek8s type=AVC msg=audit(1685993834.340:12570034): avc:  denied  { read } for  pid=917844 comm="cat" name="name" dev="dm-7" ino=1074634823 scontext=system_u:system_r:container_t:s0:c289,c356 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file permissive=0

[dajester2013]

Some more debugging:

1. Modified rke2/agent/pod-manifests/etcd.yaml to run an alpine pod in continuous loop in order to get a shell:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    etcd.k3s.io/initial: '{"initial-advertise-peer-urls":"https://10.112.35.155:2380","initial-cluster":"simplek8s-124be28a=https://10.112.35.155:2380","initial-cluster-state":"new"}'
  creationTimestamp: null
  labels:
    component: etcd
    tier: control-plane
  name: etcd
  namespace: kube-system
  uid: e18aa5e5b83a5a3c56d78e4054612394
spec:
  containers:
  - image: alpine
    command: [ "sh", "-c", "--" ]
    args: [ "while true; do sleep 30; done;" ]
    #args:
    #- --config-file=/var/lib/rancher/rke2/server/db/etcd/config
    #command:
    #- etcd
    env:
    - name: FILE_HASH
      value: 704ad7c155958632a443d961cde068d1fd4084accbc5c6202d79cc91bca2614a
    - name: NO_PROXY
      value: .svc,.cluster.local,10.42.0.0/16,10.43.0.0/16
    - name: POD_HASH
      value: 5aa33fe3f1ca84a711c7fce7308fbc49
#    image: index.docker.io/rancher/hardened-etcd:v3.5.7-k3s1-build20230406
...

[root@simplek8s rke2]# crictl ps
CONTAINER           IMAGE               CREATED              STATE               NAME                ATTEMPT             POD ID              POD
bcd92941412e6       5e2b554c1c45d       About a minute ago   Running             etcd                69                  60dda8aa93269       etcd-simplek8s

2. Shell into alpine container.

[root@simplek8s rke2]# crictl exec -ti bcd sh

3. Any command ran resulted in Permission Denied

~ $ ls
sh: ls: Permission denied
~ $ id
sh: id: Permission denied

[dajester2013]

Some more debugging, the missing X bit on the parent directories prevents etcd user from accessing contents of it's folder using bash:

[root@simplek8s db]# sudo -u etcd cat /var/lib/rancher/rke2/server/db/etcd/config
cat: /var/lib/rancher/rke2/server/db/etcd/config: Permission denied
[root@simplek8s db]# chmod o+x /var/lib/rancher/; chmod o+x /var/lib/rancher/rke2/; chmod o+x /var/lib/rancher/rke2/server; chmod o+x /var/lib/rancher/rke2/server/db;
[root@simplek8s db]# sudo -u etcd cat /var/lib/rancher/rke2/server/db/etcd/config
advertise-client-urls: https://10.112.35.155:2379
client-transport-security:
  cert-file: /var/lib/rancher/rke2/server/tls/etcd/server-client.crt
  client-cert-auth: true
  key-file: /var/lib/rancher/rke2/server/tls/etcd/server-client.key
  trusted-ca-file: /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
data-dir: /var/lib/rancher/rke2/server/db/etcd
election-timeout: 5000
experimental-initial-corrupt-check: true
heartbeat-interval: 500
initial-advertise-peer-urls: https://10.112.35.155:2380
initial-cluster: simplek8s-1725bd30=https://10.112.35.155:2380
initial-cluster-state: new
listen-client-urls: https://127.0.0.1:2379,https://10.112.35.155:2379
listen-metrics-urls: http://127.0.0.1:2381
listen-peer-urls: https://127.0.0.1:2380,https://10.112.35.155:2380
log-outputs:
- stderr
logger: zap
name: simplek8s-1725bd30
peer-transport-security:
  cert-file: /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.crt
  client-cert-auth: true
  key-file: /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.key
  trusted-ca-file: /var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt
snapshot-count: 10000

However, the pod is still not starting.

[brandond]

Has the error changed?

What caused you to restrict the permissions on those directories beyond the defaults managed by rke2?

[dajester2013]

I didn’t restrict them, I expanded them. The default permissions were 700, I chmodded to 701.

…

On Wed, Jun 7, 2023 at 11:41 AM Brad Davidson ***@***.***> wrote: Has the error changed? What caused you to restrict the permissions on those directories beyond the defaults managed by rke2? — Reply to this email directly, view it on GitHub <#4313 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABC5FWGYFDM4MZY2U6ESVE3XKCVLPANCNFSM6AAAAAAYYUUV2E> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[dajester2013]

No change in error message in the etcd container. On Wed, Jun 7, 2023 at 11:43 AM Jesse Shaffer ***@***.***> wrote:

…

I didn’t restrict them, I expanded them. The default permissions were 700, I chmodded to 701. On Wed, Jun 7, 2023 at 11:41 AM Brad Davidson ***@***.***> wrote: > Has the error changed? > > What caused you to restrict the permissions on those directories beyond > the defaults managed by rke2? > > — > Reply to this email directly, view it on GitHub > <#4313 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABC5FWGYFDM4MZY2U6ESVE3XKCVLPANCNFSM6AAAAAAYYUUV2E> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[brandond]

Can you show what your permissions are?

On the host you should see:

root@rke2-server-1:/# for DIR in /var/lib/rancher/rke2/server/db/etcd /var/lib/rancher/rke2/server/db /var/lib/rancher/rke2/server /var/lib/rancher/rke2 /var/lib/rancher; do ls -lad $DIR; done
drwx------ 3 etcd etcd 4096 Jun  7 20:59 /var/lib/rancher/rke2/server/db/etcd
drwx------ 3 root root 4096 Jun  7 20:59 /var/lib/rancher/rke2/server/db
drwxr-xr-x 7 root root 4096 Jun  7 20:59 /var/lib/rancher/rke2/server
drwxr-xr-x 5 root root 4096 Jun  7 20:59 /var/lib/rancher/rke2
drwxr-xr-x 3 root root 4096 Jun  7 20:59 /var/lib/rancher

And in the container:

root@rke2-server-1:/# crictl exec $(crictl ps -q --name=etcd) sh -c 'for DIR in /var/lib/rancher/rke2/server/db/etcd /var/lib/rancher/rke2/server/db /var/lib/rancher/rke2/server /var/lib/rancher/rke2 /var/lib/rancher; do ls -lad $DIR; done'
drwx------ 3 999 999 4096 Jun  7 20:59 /var/lib/rancher/rke2/server/db/etcd
drwxr-xr-x 3 root root 4096 Jun  7 20:59 /var/lib/rancher/rke2/server/db
drwxr-xr-x 4 root root 4096 Jun  7 20:59 /var/lib/rancher/rke2/server
drwxr-xr-x 3 root root 4096 Jun  7 20:59 /var/lib/rancher/rke2
drwxr-xr-x 3 root root 4096 Jun  7 20:59 /var/lib/rancher

[dajester2013]

[root@simplek8s jshaffer]# rm -rf /var/lib/rancher
[root@simplek8s jshaffer]# rke2 server
...
[root@simplek8s jshaffer]# for DIR in /var/lib/rancher/rke2/server/db/etcd /var/lib/rancher/rke2/server/db /var/lib/rancher/rke2/server /var/lib/rancher/rke2 /var/lib/rancher; do ls -lad $DIR; done
drwx------. 2 etcd etcd 32 Jun  8 08:39 /var/lib/rancher/rke2/server/db/etcd
drwx------. 3 root root 18 Jun  8 08:39 /var/lib/rancher/rke2/server/db
drwx------. 7 root root 117 Jun  8 08:41 /var/lib/rancher/rke2/server
drwx------. 5 root root 56 Jun  8 08:41 /var/lib/rancher/rke2
drwx------. 3 root root 18 Jun  8 08:39 /var/lib/rancher

I cannot execute the same in the container, as it is stuck in a crash loop.

[dajester2013]

Based on your comment, I did try this:

[root@simplek8s jshaffer]# for DIR in /var/lib/rancher/rke2/server/db /var/lib/rancher/rke2/server /var/lib/rancher/rke2 /var/lib/rancher; do chmod 755 $DIR; done

[root@simplek8s jshaffer]# rke2 server

### In a separate terminal
[root@simplek8s jshaffer]# crictl logs $(crictl ps -aq --name=etcd)
{"level":"info","ts":"2023-06-08T13:12:10.047Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","--config-file=/var/lib/rancher/rke2/server/db/etcd/config"]}
{"level":"warn","ts":"2023-06-08T13:12:10.048Z","caller":"etcdmain/etcd.go:75","msg":"failed to verify flags","error":"open /var/lib/rancher/rke2/server/db/etcd/config: permission denied"}

[root@simplek8s jshaffer]# ausearch -m avc
Email option is specified but /usr/lib/sendmail doesn't seem executable.
<no matches>

[dajester2013]

Perhaps related? STIG requirements state RHEL 8 must set the umask value to 077 for all local interactive user accounts.

https://www.stigviewer.com/stig/red_hat_enterprise_linux_8/2021-06-14/finding/V-230384

[dajester2013]

OK - the STIG umask requirement is what prevented etcd from starting:

[root@simplek8s /]# rke2-killall.sh 
[root@simplek8s /]# rm -rf /var/lib/rancher
[root@simplek8s /]# umask 0000
[root@simplek8s /]# rke2 server

This allows etcd to start successfully. I can send you a recursive directory listing if that would help discover what other folders are missing the necessary permissions?

Unfortunately, I'm still having issues, as I'm now getting:

INFO[0101] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
INFO[0105] Cluster-Http-Server 2023/06/08 09:13:40 http: TLS handshake error from 127.0.0.1:57174: remote error: tls: bad certificate

[root@simplek8s rke2]# crictl logs $(crictl ps -aq --name=kube-apiserver)
I0608 14:29:19.482982       1 server.go:563] external host was not specified, using 10.112.35.155
E0608 14:29:19.484047       1 run.go:74] "command failed" err="enable-admission-plugins plugin \"PodSecurityPolicy\" is unknown"

EDIT: the above was a problem with the custom configuration I copied from the ansible playbook. Removing the line with that configuration appears to allow it to run.

[brandond]

Yeah, I was thinking that it looked like a umask problem.

That admission plugin has been removed from Kubernetes; I suspect the playbook you're working off of hasn't been updated to support the version you're using.

[dajester2013]

OK - so, initial startup using the rke2 binary on a STIG'd RHEL will result in a failed etcd container.

[brandond]

I would probably just recommend adding a UMask=0000 or perhaps UMask=0022 entry to the rke2 systemd unit, to override the nonstandard default umask enforced by your STIG hardening.

[pranaysahith]

@dajester2013 I am facing the same issue with RHEL STIG. Can you please let me know which custom configuration have you had to remove from the Ansible playbook to fix this issue?

[brandond]

if you'd prefer to leave the default system umask alone, a better idea would be to add a systemd override file to the rke2-server service, with content like:

[Service]
UMask=0022

[dajester2013]

So, I’ve not had to make that change personally, it seems like my problem stemmed from attempting to launch ‘rke2 server’ initially from an interactive terminal, instead of enabling the systemd unit. Next time I attempt to install, I will attempt using the systemd unit first as is, and if that fails, will set the UMask entry in the service file. To modify the global umask setting would be a STIG finding.

…

On Tue, Jun 13, 2023 at 7:22 PM Brad Davidson ***@***.***> wrote: if you'd prefer to leave the default system umask alone, a better idea would be to add a systemd override file to the rke2-server service, with content like: [Service] UMask=0022 — Reply to this email directly, view it on GitHub <#4313 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABC5FWBBDC7RZ46NU2RB4STXLD74HANCNFSM6AAAAAAYYUUV2E> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[brandond]

Oh, I don't think it was clear that you weren't actually starting the service.

[pranaysahith]

@brandond @dajester2013 Thanks for your reply. A fresh installation works ok when UMask is set in the rke2-server service.
When I reset the cluster using below command, it fails because of the etcd permission.

sudo /usr/local/bin/rke2 server --cluster-reset

Do you know if its possible to do cluster reset any other way without the etcd permission issue?

[dajester2013]

My suggestion is, from the terminal, temporarily modify the umask to 022 before running any rke2 commands. umask 022 rke2 ….. umask 077