Use fsGroup and handles NFS shares

[tchellomello]

Fixes: #395
Fixes: #357

This PR accomplishes a couple of things based on different environments.

When using the projects_persistence with a projects_existing_claim, depending on the storage provide a block device or filesystem device will present to the AWX instance writable only by the root user. This introduces a problem as the awx user is unable to write under /var/lib/awx/projects therefore not being able to sync projects.

Using the fsGroup address the issue for block and fs mounts so the volume will be mounted allowing the awx user to write on it. However, fsGroup does not work for NFS shares as noted here: kubernetes/examples#260

To work around this issue, we can use the initContainer approach to change the permissions and privileges for the mounted volume.

Please refer to #395 (comment) and #357 (comment) for a full write-up while troubleshooting this issue.

Another important addition is the ability to the user specify extra volumes as mentioned at #395 (comment)

For example:

spec:
  replicas: 1 
  ingress_type: none
  service_type: loadbalancer
  projects_existing_claim: awx-testing-nfs-claim
  projects_persistence: true
  extra_volumes: |
    - name: shared-volume
      persistentVolumeClaim:
        claimName: awx-testing-nfs-claim2
  init_container_extra_volume_mounts: |
    - name: shared-volume
      mountPath: /shared
  init_container_extra_commands: |
    # this will ensure the awx user can write on this volume
    chmod 775 /shared
    chgrp 1000 /shared
    date >> /var/lib/awx/projects/been_here
  task_extra_volume_mounts: |
    - name: shared-volume
      mountPath: /shared

TL;DR

Before with nfs

kubectl iexec awx /bin/bash                                                               15:08:49
Namespace: default | Pod: ✔ awx-nfs-596f6cf8b7-lb4xq
Container: ✔ awx-nfs-web
bash-4.4$ ls -lahd /var/lib/awx/projects
d---------. 1 root root 14 Jun 19 18:43 /var/lib/awx/projects
bash-4.4$ df -h /var/lib/awx/projects
Filesystem                        Size  Used Avail Use% Mounted on
192.168.111.46:/volume1/test-nfs-share  3.5T  1.3T  2.3T  36% /var/lib/awx/projects
bash-4.4$ whoami
awx
bash-4.4$ id awx
uid=1000(awx) gid=0(root) groups=0(root)
bash-4.4$ touch /var/lib/awx/projects/testing
touch: cannot touch '/var/lib/awx/projects/testing': Permission denied

Before with block

(py39) mdemello@storm ~/d/P/a/reproducer-357 (devel)> kubectl iexec awx-az /bin/bash                                       22:39:54
Namespace: default | Pod: ✔ awx-az-84dd57946f-855mn
Container: ✔ awx-az-web
bash-4.4$ df -h /var/lib/awx/projects/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        9.8G   37M  9.8G   1% /var/lib/awx/projects
bash-4.4$ ls -lah /var/lib/awx/projects/
total 24K
drwxr-xr-x 3 root root 4.0K Jun 18 02:26 .
drwxrwxr-x 1 root root 4.0K Jun 18 02:39 ..
drwx------ 2 root root  16K Jun 18 02:26 lost+found
bash-4.4$ id awx
uid=1000(awx) gid=0(root) groups=0(root)
bash-4.4$ whoami 
awx
bash-4.4$ touch /var/lib/awx/projects/test
touch: cannot touch '/var/lib/awx/projects/test': Permission denied

After with nfs

$ kubectl iexec awx /bin/bash                                                                   15:27:14
Namespace: default | Pod: ✔ awx-nfs-696dd649f5-zlqc4
Container: ✔ awx-nfs-web
bash-4.4$ df -h /var/lib/awx/projects/
Filesystem                        Size  Used Avail Use% Mounted on
192.168.111.46:/volume1/test-nfs-share  3.5T  1.3T  2.3T  36% /var/lib/awx/projects
bash-4.4$ whoami 
awx
bash-4.4$ id
uid=1000(awx) gid=0(root) groups=0(root),1000
bash-4.4$ touch /var/lib/awx/projects/testing-works-now
bash-4.4$ ls -la /var/lib/awx/projects/testing-works-now
-rw-rw-rw-. 1 awx root 0 Jun 19 19:23 /var/lib/awx/projects/testing-works-now

After with block

$ kubectl iexec awx-az /bin/bash                                                                            23:39:31
Namespace: default | Pod: ✔ awx-az-5fc5dd5767-lc48z
Container: ✔ awx-az-web
bash-4.4$ ls -lad /var/lib/awx/projects/
drwxrwsr-x 5 root 1000 4096 Jun 18 03:09 /var/lib/awx/projects/
bash-4.4$ df -h /var/lib/awx/projects/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        9.8G   37M  9.8G   1% /var/lib/awx/projects
bash-4.4$ touch /var/lib/awx/projects/test_works && echo $?
0

TODO:

• Tested @ AWS
• Tested @ OCP
• Tested @ Azure ✔️
• Tested @ vanilla k8s with hostPath
• Tested @ vanilla k8s
• Tested @ vanilla k8s with NFS provisioner

[felipe4334]

@tchellomello do you know if this changes have been merged with 0.12.0?

[tchellomello]

@felipe4334 no, it's not in the 0.12.0. I'll finish testing this PR in some other environment and flip to get an official review. It should be included on the next version. 🤞

[wohnout]

Thanks @tchellomello. This pull request helps a lot !