Read-only filesystem issue - hazelcast exporter - 8.3.0

[inghak]

Describe the bug

I am adding hazelcast exporter the "Adding dynamic exporters to Zeebe Broker" - way, using Helm charts into Kubernetes. This works in 8.2.9 and not in 8.3.0.

The error is when zeebe pod starts, it logs: mkdir: cannot create directory ‘/usr/local/zeebe/exporters/’: Read-only file system.

I understand this has to do with the 8.3.0 change to no longer run as root user 0, but rather non-root user 1000. I have tried to fix this, but am struggelig to make it work.

I have done the suggested changes in the upgrade guide:

zeebe:
  podSecurityContext:
    fsGroup: 1000

and alternatively:

zeebe:
  containerSecurityContext:
    runAsUser: 0

And also add to the initcontainer config:

    securityContext:
      runAsNonRoot: true
      runAsUser: 1000

I have also tried to add the /usr/local/zeebe/exporters folder as an empty folder with extraVolumeMounts but then I get error from zeebe pod : already exists in place of read-only error.

I have experimented with various flavours to get this to work.

The relevant parts of the zeebe configuration:

zeebe:
  ...
  pvcSize: 10Gi
  pvcStorageClassName: standard
  pvcAccessModes:
   - ReadWriteOnce
  extraInitContainers:
  - name: init-exporters-hazelcast
    image: busybox:1.36.1
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
    command: ['/bin/sh', '-c']
    args: [
      'wget --no-check-certificate https://github.com/camunda-community-hub/zeebe-hazelcast-exporter/releases/download/1.4.0/zeebe-hazelcast-exporter-1.4.0-jar-with-dependencies.jar -O /exporters/zeebe-hazelcast-exporter.jar',
    ]
    volumeMounts:
    - name: exporters
      mountPath: exporters/
  env:
  - name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_JARPATH
    value: exporters/zeebe-hazelcast-exporter.jar

...

To Reproduce

To reproduce, add hazelcast integration using extraInitContainer with Helm 8.2.9 sucessfully, and then do the same with 8.3.0.

Expected behavior

The hazelcast integration should work after upgrade to 8.3.0 and adding the fsGroup: 1000.

Log/Stacktrace

Full Stacktrace

<STACKTRACE>

Environment:

• OS: Linux
• Zeebe Version: 8.3.0
• Configuration: Zeebe, Hazelcast

[jessesimpson36]

I think your mountPath needs to have the full filepath: /usr/local/zeebe/exporters, not the relative path exporters

[inghak]

Unfortunately changing the mounthPath as suggested does not help. The origial syntax works in 8.2.9 and stop working in 8.3.x.
It is the zeebepod that has a problem. The hazelcast container is up and running ok.

This is the log for the zeebepod:

+ export ZEEBE_BROKER_CLUSTER_NODEID=0
+ ZEEBE_BROKER_CLUSTER_NODEID=0
++ ls -A /exporters/
+ '[' zeebe-hazelcast-exporter.jar ']'
+ mkdir /usr/local/zeebe/exporters/
mkdir: cannot create directory ‘/usr/local/zeebe/exporters/’: Read-only file system

Have any of you Zeebe guys tried to mount hazecast like this in 8.3?

[inghak]

I think I found a solution myself. In addition to adding:

securityContext:
      runAsNonRoot: true
      runAsUser: 1000

to the extraInitContainer, I also had to add this to the zeebe section:

containerSecurityContext:
    readOnlyRootFilesystem: false

👍

[aabouzaid]

@inghak Disabling security is definitely not the right way to make it.

The issue comes from this start-up script, where it copies all files from /exporters (which is a tmp path) to Zeebe exporters path /usr/local/zeebe/exporters (it was enabled for legacy reasons where users use Zeebe image as a base and add exporters to it).

So, to fix that issue, there are 2 things to do,

First, mount a new vol under /usr/local/zeebe/exporters.
Second, ensure that the downloaded exporters are in the correct place.

It should be like this:

zeebe:
...
  extraVolumes
  - name: exporters-zeebe
    emptyDir: {}
  extraVolumeMounts:
  - name: exporters-zeebe
    mountPath: /usr/local/zeebe/exporters

Yet, that will not work because mkdir will fail in that case because the dir already exists (it should be at lease mkdir -p ...).

So, I'd consider that as a bug.

Probably we will just add that -p in 8.3.x, and in 8.4.0 will get rid of the 2-step copy of the exporters.

[aabouzaid]

Fixed by #1080, and it will be part of the next release
v8.3.2 (feel free to open this issue if it still doesn't work).