Add support for GKE Autopilot

[dorkamotorka]

Is your feature request related to a problem? Please describe.
The problem is that, currently promtail (default) values.yml configuration uses HostPaths docker volumes: /run/promtail, /var/lib/docker/containers, /var/log/pods but the GKE (with its default security) Autopilot only allows HostPaths mounts under /var/log. So when I try to deploy the promtail onto the GKE Autopilot I get:

Error: admission webhook "gkepolicy.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.
Violations details: {"[denied by autogke-no-write-mode-hostpath]":["hostPath volume run in container promtail is accessed in write mode; disallowed in Autopilot.","hostPath volume containers used in container promtail uses path /var/lib/docker/containers which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/]."]}
Requested by user: 'terraform@pw-infra-common.iam.gserviceaccount.com', groups: 'system:authenticated'.

And this is the promtail default configuration, where it specifies the hostPaths:

defaultVolumes:
  - name: run
    hostPath:
      path: /run/promtail
  - name: containers
    hostPath:
      path: /var/lib/docker/containers
  - name: pods
    hostPath:
      path: /var/log/pods

Describe the solution you'd like
I believe the issue should be resolved between Grafana and GCP to support this.

Describe alternatives you've considered
Alternative is probably to deploy Promtail outside of the GKE cluster, but this is undesired.

[dorkamotorka]

Note: that this cannot be easily resolved by just changing the directories to use some subdirectory under /var/log because it is only allowed to be accessed in read-mode (and NOT write-mode as loki requires)

[plechi]

In GKE Autopilot clusters, only /var/log/pods is needed to get the workload logs.

• /var/lib/docker/containers is not needed as Autopilot clusters don't support docker.
• /run/promtail should be optimally a shared volume, however it works (but might send duplicates on restart) if you define it as emptyDir

If you want to get other logs (like Autopilot system logs), you can use Google PubSub instead of the directory scraping of Promtail.

So in the promtail helm values, set:

defaultVolumeMounts:
  - name: pods
    mountPath: /var/log/pods
    readOnly: true
  - name: run
    mountPath: /run/promtail

defaultVolumes:
  - name: pods
    hostPath:
      path: /var/log/pods
  - name: run
    emptyDir: { }

[marcoboffi]

In GKE Autopilot clusters, only /var/log/pods is needed to get the workload logs.

• /var/lib/docker/containers is not needed as Autopilot clusters don't support docker.
• /run/promtail should be optimally a shared volume, however it works (but might send duplicates on restart) if you define it as emptyDir

If you want to get other logs (like Autopilot system logs), you can use Google PubSub instead of the directory scraping of Promtail.

So in the promtail helm values, set:

defaultVolumeMounts:
  - name: pods
    mountPath: /var/log/pods
    readOnly: true
  - name: run
    mountPath: /run/promtail

defaultVolumes:
  - name: pods
    hostPath:
      path: /var/log/pods
  - name: run
    emptyDir: { }

So does this configuration work on GKE Autopilot? Can I set podSecurityContext with default values ?
podSecurityContext:
runAsUser: 0
runAsGroup: 0