Try out GKE Autopilot

[sroettger]

https://cloud.google.com/blog/products/containers-kubernetes/introducing-gke-autopilot

[sirdarckcat]

https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#host_options_restrictions

gcsfuse uses hostpath mapping in write mode.

[sroettger]

we had an alternative to hostpath but that was too expensive, right?

…

On Fri, Mar 19, 2021 at 7:55 AM sirdarckcat ***@***.***> wrote: https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#host_options_restrictions gcsfuse uses hostpath mapping in write mode. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#259 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEM2JHSK7CBES6VGAUOQ4JTTELYPPANCNFSM4ZB3R2IA> .

[sirdarckcat]

it also uses privileged https://github.com/google/kctf/blob/v1.0/kctf-operator/pkg/resources/daemon-gcsfuse.go - can we use fuse without priv?

[sirdarckcat]

https://lwn.net/Articles/742138/?

[sirdarckcat]

docker/for-linux#321

[sirdarckcat]

to use fuse without priv we would need this: moby/moby#41880

apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo-deployment
  labels:
    app: foo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: foo
  template:
    metadata:
      labels:
        app: foo
      annotations:
        container.apparmor.security.beta.kubernetes.io/foo: unconfined
    spec:
      containers:
      - name: foo
        image: ubuntu:latest
        command: ['/bin/bash']
        args: ["-c", "sleep 1d"]
        securityContext:
          procMount: UnmaskedProcMount
          seccompProfile:
            type: Unconfined
        volumeMounts:
          - name: dev-dir
            mountPath: /dev
      volumes:
        - name: dev-dir
          hostPath:
            path: /dev

I guess we need to choose between autopilot and gcsfuse.

[sroettger]

In that case we should stick with gcsfuse unless we find some good alternative. Maybe revisit in 6-12 months?

…

On Fri, Mar 19, 2021 at 12:19 PM sirdarckcat ***@***.***> wrote: to use fuse without priv we would need this: moby/moby#41880 <moby/moby#41880> apiVersion: apps/v1kind: Deploymentmetadata: name: foo-deployment labels: app: foospec: replicas: 1 selector: matchLabels: app: foo template: metadata: labels: app: foo annotations: container.apparmor.security.beta.kubernetes.io/foo: unconfined spec: containers: - name: foo image: ubuntu:latest command: ['/bin/bash'] args: ["-c", "sleep 1d"] securityContext: procMount: UnmaskedProcMount seccompProfile: type: Unconfined volumeMounts: - name: dev-dir mountPath: /dev volumes: - name: dev-dir hostPath: path: /dev I guess we need to choose between autopilot and gcsfuse. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#259 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEM2JHUBN5QPNAVMBPFFMCTTEMXM3ANCNFSM4ZB3R2IA> .

[sirdarckcat]

lets do that, yea. I'll leave this open.

[eljeffeg]

Curious if there were any updates or progress on this? Autopilot looks pretty cool for a Kubernetes newb.

[sirdarckcat]

it is really cool! it doesn't completely work due to our use of hostPath, so we would most likely need to find another way to do persistent storage.

…

[eljeffeg]

I'm not sure if it's the same issue, but I had done some research a bit ago with regard to ephemeral disks with Heroku. One consideration was something like s3fuse and s3monkey. Expect something similar could be done for Google Cloud.

[sirdarckcat]

I think they use hostPath, yes

[eljeffeg]

It was suggested in GoogleCloudPlatform/gcsfuse#497 that csi-gcs could be used.

[sirdarckcat]

Ah yes, that would work!

…

On Fri, 29 Oct 2021, 15:30 ElJeffe, ***@***.***> wrote: It was suggested in GoogleCloudPlatform/gcsfuse#497 <GoogleCloudPlatform/gcsfuse#497> that csi-gcs <https://github.com/ofek/csi-gcs> could be used. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#259 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAICQIMNWANN7PAO4FVKEDUJKOVVANCNFSM4ZB3R2IA> .

[eljeffeg]

Holly $h**, KCTF in the Cloud is expensive. I had started the cloud walkthrough a couple weeks ago. Didn't realize that Google Cloud charges you for spinning it up. Got a bill today and it wasn't cheap. There should be a warning on the setup for those Compute Engines. Ouch

[sirdarckcat]

Oh, submit a ticket. Cloud Support might refund you.

[sirdarckcat]

It appears autopilot doesn't allow pods with SYS_ADMIN capability to be admitted, so we wouldn't be able to isolate tasks.

[sirdarckcat]

(context: sys_admin is required for using clone(new_user) and the default seccomp profile requires sys_admin to call clone(new_user))

[sirdarckcat]

Ah we can change the default policy