Limit required privileges to run some MetalK8s control plane and infra services

[thomasdanan]

Component:

Why this is needed:

Some MetalK8s users may have security constraints when it comes to run MetalK8s. Limiting the number of services requiring root user to run will help to improve MetalK8s adoption.

it likely makes sense for us to not run several things that are running as root today as non-root. Some are using a non-0 UID already (using some randomly chosen UID). However, we should for all node-local services (i.e., static manifests) create the appropriate user(s) and use their UID in the Pod manifest. Of course, we then need to ensure the applicable user(s) has the right to read e.g. x509 keys and such.

Examples of things we can likely run with lower privileges:

• Repository nginx (no need to bind to any low ports)
• apiserver-proxy nginx (same as above)
• kube-scheduler, kube-apiserver, kube-controller-manager
• etcd, but then we must properly manage the ownership of /var/lib/etcd
• salt-api, maybe salt-master, if proper permissions on the sockets and such in /var/run/salt can be set

Furthermore, we can use securityContexts to further drop capabilities (and some more), such that services that do (need to) run as root are still somewhat constrained.

One kubeadm issue relates to this: kubernetes/kubeadm#1367

What should be done:

Implementation proposal (strongly recommended):

Test plan: