Permission errors when deploying on K8S

[bjwschaap]

When deploying the container on Kubernetes 1.8.6, and use a persistent volume, the container fails starting with:

/usr/share/container-scripts/postgresql/common.sh: line 127: /var/lib/pgsql/passwd: Permission denied

To reproduce:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-db-postgresql-data
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: test-db
  labels:
    app: test-db
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-db
  template:
    metadata:
      labels:
        app: test-db
    spec:
      containers:
      - name: test-db
        image: registry.centos.org/postgresql/postgresql:9.6
        imagePullPolicy: IfNotPresent
        env:
        - name: POSTGRESQL_ADMIN_PASSWORD
          value: secret
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - exec pg_isready --host $POD_IP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 6
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - exec pg_isready --host $POD_IP
          initialDelaySeconds: 20
          timeoutSeconds: 3
          periodSeconds: 5
        ports:
        - containerPort: 5432
          protocol: TCP
        volumeMounts:
        - mountPath: /var/lib/pgsql
          name: test-db-postgresql-data
          readOnly: false
      volumes:
      - name: test-db-postgresql-data
        persistentVolumeClaim:
          claimName: test-db-postgresql-data

I have

> kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-09T21:51:54Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:23:29Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
> kubectl get nodes
ip-172-20-x-x.eu-west-1.compute.internal   Ready     master    23h       v1.8.6
ip-172-20-y-y.eu-west-1.compute.internal   Ready     node      23h       v1.8.6
ip-172-20-z-z.eu-west-1.compute.internal    Ready     node      23h       v1.8.6
ip-172-20-a-a.eu-west-1.compute.internal    Ready     master    23h       v1.8.6
ip-172-20-b-b.eu-west-1.compute.internal    Ready     node      23h       v1.8.6
ip-172-20-c-c.eu-west-1.compute.internal     Ready     master    23h       v1.8.6

3 masters, 3 nodes, 1 region, 3 AZs, each AZ has 1 master and 1 node in HA mode.
RBAC is enabled on the cluster.

[bjwschaap]

Running a shell in the container shows:

bash-4.2$ ls -al /var/lib/pgsql/
total 24
drwxrwx---  5 postgres root 4096 Mar  1 09:52 .
drwxr-xr-x 13 root     root 4096 Mar  1 09:52 ..
-rwxrw----  3 postgres root  267 Feb 28 17:47 .bash_profile
drwxrwx---  3 postgres root 4096 Mar  1 09:52 .pki
drwxrwx---  4 postgres root 4096 Mar  1 09:52 9.6
drwxr-xr-x  4 root     root 4096 Mar  1 07:20 data
bash-4.2$ ls -al /var/lib/pgsql/data
total 28
drwxr-xr-x 4 root     root  4096 Mar  1 07:20 .
drwxrwx--- 5 postgres root  4096 Mar  1 09:52 ..
drwxr-xr-x 2 root     root  4096 Mar  1 07:20 data
drwx------ 2 root     root 16384 Mar  1 07:20 lost+found
bash-4.2$ ls -al /var/lib/pgsql/data/data/
total 8
drwxr-xr-x 2 root root 4096 Mar  1 07:20 .
drwxr-xr-x 4 root root 4096 Mar  1 07:20 ..

This shows the volume mounted as root, and no write permissions for group/other.

[bjwschaap]

I guess, there's no way to solve this from within the container. Adding:

securityContext:
        fsGroup: 0

To my deployment/container spec fixed the problem.