Replace ownership change init container with K8s FSGroup

[tangledbytes]

Explain the changes

This PR attempts to replace FS ownership change init container with K8s FSGroup in an attempt to:

1. Fix Bug: failing to change ownership of the NFS based PVC for PostgreSQL pod by using kube_pv_chown utility noobaa-core#7030.

FSGroup unfortunately is not a silver bullet:

• Its usage helps us gets rid of init container but that just delegates the responsibility of changing permissions and setting owner to kubelet. In theory, this should be more performant because CRI doesn't have to spin up a container just so that we can do our permission adjustment rather kubelet does that automatically on the mounted volume.
• Kubelet will perform 2 actions on the mounted directory
  • Change the group of the owner to the fsGroup passed in pod spec. Kubelet does this simply by running os.Lchown(file, -1, int(*fsGroup)) which is run as a root user. This should work most of the time HOWEVER in the issue linked above, it WILL fail because any operation performed by the root user will be executed as nobody:nogroup by NFS. This does NOT causes any issues however because kubelet just logs the error instead of erroring out. This works out in our favor in THIS case.
  • Change the directory permissions to 0660 | os.ModeSetgid | 0110 which ensures that we get drwxdrwx-- permission on the mounted directory which is good enough for us because we run group ID 0 while os.ModeSetgid set the first octal to 4 implying that directories and files created within this directory will inherit the group ownership. This suffers the same problem in NFS (root_squash) mounts but because kubelet will not error out, it should work just fine.

So this does NOT exactly fixes out problems with NFS mounts but makes sure that our DB pod doesn't ends up in CLBO because of permission issues but will crash if our second init container fails to do its thing (which it will if NFS mount does not provide enough permissions - which I think are available in the above issue's environment)

Reference: Kubernetes FSGroup Source Code

Tests Performed

1. Deploy and run in fresh Minikube Cluster.
2. Deploy and run in fresh OpenShift Cluster (AWS gp2 storageclass).
3. Deploy and run in fresh OpenShift Cluster (AWS NFS mount with drwxdrwxdrwx mount permissions and root_squash enabled).

Issues: Fixed #xxx / Gap #xxx

Fixes noobaa/noobaa-core#7030.

Testing Instructions:

Passing pre-existing CLI tests should be good enough indication as only deployment has changed.

[baum]

Hello 👋 Ukarsh,

You have described test scenarios verbosely, however all reuse the existing PV. I'm wondering about clean install scenario in OpenShift environment. Is DB pod without ownership change init container able to use a fresh PV which was just provisioned for it?

For instance, for the "2. Normal NooBaa deployment in OpenShift" what happens if we change "ii. Delete the DB Pod." to "ii. Delete the DB Pod and PVC" making sure DB gets a fresh PV?

[tangledbytes]

Hi @baum! That's a good test! I haven't performed it yet. Lemme get back to you once I am done testing this. Please lemme know if there are more tests that I should perform here.

[tangledbytes]

Under development - ignore for review ATM.

[tangledbytes]

I don't think that golangci-lint is upset with the PR rather something related to actions/runner-images#7188? 🤔

[baum]

I don't think that golangci-lint is upset with the PR rather something related to actions/runner-images#7188? 🤔

Please rebase, hope #1088 fixes the golangci-lint issue

[tangledbytes]

Thank you @baum! It fixed the test.