Kubernetes Examples?

[sfxworks]

I tried to write up an example, but even witht he right gid/uid, I can't get this working correctly. I am thinking I am missing a factor when it comes to mount prop + fuse and have to use the host directly or apply a daemonset w/ a bidirectional mount, then have the pod reference that via hostpath.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: goofys-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: goofys-nginx
  template:
    metadata:
      labels:
        app: goofys-nginx
    spec:
      containers:
      - image: my.private.registry/lib/goofys
        securityContext:
          privileged: true
        imagePullPolicy: Always
        name: goofys
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: s3
                key: id
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef: 
                name: s3
                key: secret
        command:
        - /bin/sh
        args:
          - -c 
          - ./goofys -f --endpoint https://ceph.object.storage/ --region lax1 --uid 101 --gid 101 --dir-mode 0777 --file-mode 0777 bucket1 /mnt/www 
        volumeMounts:
          - name: www
            mountPath: /mnt/www
            mountPropagation: Bidirectional
      - image: nginx:latest
        imagePullPolicy: Always
        name: nginx
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources:
          limits:
            cpu: 750m
            memory: 512Mi
          requests:
            cpu: 250m
            memory: 100Mi
        volumeMounts:
          - name: www
            mountPath: /usr/share/nginx/html
      volumes:
      - name: www
        emptyDir: {}
      imagePullSecrets:
      - name: regcred

goofys

/mnt/www # ls -lah
total 9K
drwxrwxrwx    2 101      101         4.0K Jan  2 21:35 .
drwxr-xr-x    1 root     root        4.0K Jan  2 21:35 ..
-rwxrwxrwx    1 101      101            5 Jan  2 16:08 index.html
-rwxrwxrwx    1 101      101            7 Jan  2 21:01 test.html
-rwxrwxrwx    1 101      101            0 Jan  2 16:08 test.txt

nginx

/usr/share/nginx # ls -lah
total 12K
drwxr-xr-x    3 root     root        4.0K Nov 20 01:16 .
drwxr-xr-x    1 root     root        4.0K Nov 20 01:16 ..
drwxrwxrwx    2 nginx    nginx       4.0K Jan  2 21:35 html
/usr/share/nginx # cd html
/usr/share/nginx/html # ls -lah
total 9K
drwxrwxrwx    2 nginx    nginx       4.0K Jan  2 21:35 .
drwxr-xr-x    3 root     root        4.0K Nov 20 01:16 ..
-rwxrwxrwx    1 nginx    nginx          5 Jan  2 16:08 index.html
-rwxrwxrwx    1 nginx    nginx          7 Jan  2 21:01 test.html
-rwxrwxrwx    1 nginx    nginx          0 Jan  2 16:08 test.txt

I've tried tweaking fsGroup and swapping nginx for apache as well. Even though the gid/uid matches, it still yields permission issues.
Does anyone have an example/had success with this? My initial manifests was based on cloudposse/docs#84

I want to add tweaks to it down the line before donating one via PR. Such as referencing the exact permission needed similar in s3fs-fuse/s3fs-fuse#647 (comment) and such.

[kahing]

you probably need -o allow_other as well