Improve scenario with running Liberty containers with readOnlyRootFilesystem

[leochr]

When Liberty containers are run with readOnlyRootFilesystem security context set to true, it encounters permission issues due to files being generated at runtime. In particular at these locations:

• /opt/ol/wlp/output
• /opt/ol/wlp/usr/servers/defaultServer/configDropins

It's possible to use init containers and volume mounts to get around this. But investigate whether anything can be done at container image level to improve this scenario.

[DasJayYa]

Yeah would love to see improvements in this space.

@leochr would love to see a sample on how you worked around the problem using Init Containers.

[leochr]

@DasJayYa Here are the details on the workaround. As files can be written to volumes with readOnlyRootFileSystem, the basic idea is to attach an emptyDir volume type (which will be shared among the containers) and add the locations to write to as subpath and then mount them. The init container commands copy the initial set of files the image comes with (i.e. cache, server config)

      initContainers:
        - name: wlp-init
          command:
            - /bin/bash
          args:
            - '-c'
            - "cp -rvn /opt/ol/wlp/output /wlp/\n\tcp -rvn /opt/ol/wlp/usr/servers/defaultServer/configDropins /wlp/\n\t"
          volumeMounts:
            - name: scratch
              mountPath: /wlp/output
              subPath: wlp-output
            - name: scratch
              mountPath: /wlp/configDropins
              subPath: wlp-config-dropins
      containers:
        - name: app
          env:
            - name: LOG_DIR
              value: /output/logs
          volumeMounts:
            - name: scratch
              mountPath: /tmp
              subPath: tmp
            - name: scratch
              mountPath: /opt/ol/wlp/output
              subPath: wlp-output
            - name: scratch
              mountPath: /opt/ol/wlp/usr/servers/defaultServer/configDropins
              subPath: wlp-config-dropins
      volumes:
        - name: scratch
          emptyDir: {}

[idlewis]

Liberty will write to certain locations while running, so in order to run with readOnlyRootFilesystem, some additional writeable storage is necessary. This can be mounted in the container. The difficulty is where data from the container filesystem is read from a location that is also written to.

There are two locations dealt with by the above workaround (i.e. /opt/ol/wlp/output and /opt/ol/wlp/usr/servers/defaultServer/configDropins)

For configDropins

• this location is written to at container runtime by the liberty container startup scripts which generate config xml files for the keystore and truststore (amongst others)
• it is also likely to contain xml config that was generated at container build time, either by the user or by liberty's build scripts

One possible way around this would be to add an additional 'include' location into the server.xml. The container startup scripts could write into this additional include location, which would need to be mounted storage. However, this wouldn't allow us to use config overrides at runtime.

The alternative would be to use a formalized version of the above workaround.

[idlewis]

For /opt/ol/wlp/output
This contains:
/opt/ol/wlp/output/workarea, which is populated at container build time, and contributes to startup performance.
Mounting writeable storage on top of this would be possible, but would slow down container startup.
I'm not aware of any configuration that can be used to modify this location.
/opt/ol/wlp/output/resources which might contain application resources needed at runtime
/opt/ol/wlp/output/logs, which must be writable at runtime.

I can't currently see an easy solution to making this readonly

[idlewis]

As mentioned in #457 /tmp also needs to be writeable.
This is because the /opt/ol/wlp/bin/server (the liberty startup bash script) uses heredoc. On bash < 5.1, this causes a file to be written to /tmp.

• This isn't a problem in the Java 21 base Liberty images (they use UBI 9 which uses bash 5.1)
• It could be fixed by modifying the server statup script so that it doesn't use heredoc
• There shouldn't be an issue in mounting some empty storage at /tmp to make it writeable.