System.UnauthorizedAccessException running EventStore 20.6.1 on Kubernetes

[kollerdroid]

The same error in Kubernetes StatefulSet:

System.UnauthorizedAccessException: Access to the path 'var/lib/eventstore/writer.chk' is denied.
// ...

Tested images:

• eventstore/eventstore:latest
• eventstore/eventstore:20.6.1-buster-slim

 # ...
apiVersion: apps/v1
kind: StatefulSet
 # ...
          volumeMounts:
            - name: my-eventstore-data
              mountPath: /var/lib/eventstore
 # ...
  volumeClaimTemplates:
    - metadata:
        name: my-eventstore-data
      spec:
        accessModes: ['ReadWriteOnce']
        resources:
          requests:
            storage: 5Gi
 # ...

Behind the default StorageClass there is a dynamic storage provisioner (csi.vsphere.vmware.com), which works fine.

Originally posted by @kollerdroid in #2706 (comment)

[kristapsstrals]

Had a similar issue when deploying ES on K8s. Figured that this was due to ES v20.x docker container running as non-root user by default (as opposed to older ES 5.x docker images). To fix this, you need to specify a security context on k8s pod in the stateful set:

apiVersion: apps/v1
kind: StatefulSet
spec:
  template:
    spec:
      # Allow non-root user to access PersistentVolume
      securityContext:
        fsGroup: 1000

[alenagy]

I cannot get around this issue. Tried with securityContext, with privileged, with initContainers changing ownership of the folders to eventstore user (1000). Nothing.
Tried images eventstore/eventstore:21.2.0-buster-slim and eventstore/eventstore:21.6.0-buster-slim. Not working.
Any ideas as to how to get a single pod of eventstore working on k8s?.

[danhawkins]

I also have the same issue :(

[alexeyzimarev]

For me, it was the security context. This one works in our env.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: acl-eventstore
  namespace: subs
  labels:
    app: acl
    component: acl-eventstore
    release: production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: acl
      component: acl-eventstore
      release: production
  template:
    metadata:
      namespace: subs
      labels:
        app: acl
        component: acl-eventstore
        release: production
    spec:
      volumes:
        - name: esdb-config
          configMap:
            name: acl-esdb-config
            defaultMode: 420
        - name: esdb-data
          persistentVolumeClaim:
            claimName: acl-esdb-data
      containers:
        - name: acl-eventstore
          image: 'eventstore/eventstore:21.2.0-buster-slim'
          ports:
            - name: eventstore
              containerPort: 2113
              protocol: TCP
          volumeMounts:
            - name: esdb-config
              mountPath: /etc/eventstore
            - name: esdb-data
              mountPath: /var/lib/eventstore
          livenessProbe:
            httpGet:
              path: health/live
              port: 2113
              scheme: HTTP
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: health/live
              port: 2113
              scheme: HTTP
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 60
      securityContext:
        fsGroup: 1000
  serviceName: esdb

[alenagy]

Unfortunately its still not working for me. Setting securityContext at the container level or at the spec level didn't changed the error.

[alexeyzimarev]

My cluster is GKE. The original message says it's VSphere CSI provider. Every CSI provider is different. If you use the NFS provider, or a local hostpath, you get a volume with permissions set by the OS, which none of the workloads could use. Those workloads that run as root can probably run ok, but no one wants those workloads as they are insecure.

It's hard to impossible to diagnose a specific persistent volume issue without having access to the cluster. I would suggest talking to the kubernetes cluster admin, as it's not ESDB issue, it just needs access to the volume, that's all.

[alenagy]

What's weird for me is that I also have a RabbitMQ running there and was able to solve the access issue assigning permissions on the volume folders to the rabbitmq user.
If it's an issue with my AKS cluster and volume permissions then why was RabbitMQ able to work with it but not ESDB?.

If it helps at all, here's an image of the container details. No persistent volume declared on the cluster, only mounted on the statefulset and they both have Read/Write

[alenagy]

And here's the YAML of my StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: eventstore-dev
  namespace: dev
spec:
  replicas: 1
  serviceName: eventstore-dev
  selector:
    matchLabels:
      app: eventstore-dev
  template:
    metadata:
      labels:
        app: eventstore-dev
    spec:
      containers:
      - name: eventstore-dev
        image: eventstore/eventstore:21.2.0-buster-slim
        ports:
          - containerPort: 1113
          - containerPort: 2113
        env:
          - name: EVENTSTORE_CLUSTER_SIZE
            value: "1"
          - name: EVENTSTORE_RUN_PROJECTIONS
            value: All
          - name: EVENTSTORE_START_STANDARD_PROJECTIONS
            value: "true"
          - name: EVENTSTORE_EXT_TCP_PORT
            value: "1113"
          - name: EVENTSTORE_EXT_HTTP_PORT
            value: "2113"
          - name: EVENTSTORE_INSECURE
            value: "true"
          - name: EVENTSTORE_ENABLE_EXTERNAL_TCP
            value: "true"
          - name: EVENTSTORE_ENABLE_ATOM_PUB_OVER_HTTP
            value: "true"
        volumeMounts:
        - mountPath: /var/lib/eventstore
          name: eventstore-dev-data
        - mountPath: /var/log/eventstore
          name: eventstore-dev-logs
      securityContext:
        fsGroup: 1000
      volumes:
        - name: eventstore-dev-data
          hostPath:
            path: /data/eventstore-dev
            type: DirectoryOrCreate
        - name: eventstore-dev-logs
          hostPath:
            path: /logs/eventstore-dev
            type: DirectoryOrCreate

[alexeyzimarev]

Where do you actually run it? Is it Container Apps? Having hostPath for the volumes confuses me. In managed Kubernetes you won't use hostPath.

A hostPath PersistentVolume uses a file or directory on the Node to emulate network-attached storage.

In a production cluster, you would not use hostPath. Instead a cluster administrator would provision a network resource like a Google Compute Engine persistent disk, an NFS share, or an Amazon Elastic Block Store volume.

It's clearly a local node filesystem access issue. If you use the normal network-attached volume, it should work.

[alexeyzimarev]

Closing due to lack of feedback.

[virivigio]

gcloud compute ssh bastion-host --tunnel-through-iap -- -L 12113:10.10.2.18:2113