Graylog chart fails to start - chown: changing ownership of '/usr/share/graylog/data/journal': Operation not permitted

[juliohm1978]

Is this a request for help?: No

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes:

• Helm v2.13.1
• Kubernetes v1.14.1

Which chart: stable/graylog

What happened:
Graylog fails to start because Init Containers provided by the chart do not correctly adjust file permissions in some cases.

More specifically, when PVs are automatically provisioned by a storage class (such as OpenEBS), the location of the PV on disk is inaccessible unless mounted in a container. This makes it difficult to workaround, since it is not possible to chown the PV data from outside the runtime.

The graylog container will start after its initContainer, but fail nonetheless because its user cannot chown the journal directory:

chown: changing ownership of '/usr/share/graylog/data/journal': Operation not permitted
Current master is 
Launching graylog-0 as master
pod/graylog-0 labeled
Starting graylog
Graylog Home /usr/share/graylog
Graylog User graylog
JVM Options -Djava.net.preferIPv4Stack=true -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Xms1024g -Xmx1024g
2019-04-27 04:05:49,687 INFO    [CmdLineTool] - Loaded plugin: AWS plugins 3.0.1 [org.graylog.aws.AWSPlugin] - {}
2019-04-27 04:05:49,693 INFO    [CmdLineTool] - Loaded plugin: Collector 3.0.1 [org.graylog.plugins.collector.CollectorPlugin] - {}
2019-04-27 04:05:49,694 INFO    [CmdLineTool] - Loaded plugin: Threat Intelligence Plugin 3.0.1 [org.graylog.plugins.threatintel.ThreatIntelPlugin] - {}
2019-04-27 04:05:49,788 ERROR   [CmdLineTool] - Invalid configuration - {}
com.github.joschi.jadconfig.ValidationException: Parent directory /usr/share/graylog/data/journal for Node ID file at /usr/share/graylog/data/journal/node-id is not writable
	at org.graylog2.Configuration$NodeIdFileValidator.validate(Configuration.java:302) ~[graylog.jar:?]
	at org.graylog2.Configuration$NodeIdFileValidator.validate(Configuration.java:284) ~[graylog.jar:?]
	at com.github.joschi.jadconfig.JadConfig.validateParameter(JadConfig.java:215) ~[graylog.jar:?]
	at com.github.joschi.jadconfig.JadConfig.processClassFields(JadConfig.java:148) ~[graylog.jar:?]
	at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:99) ~[graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:351) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.readConfiguration(CmdLineTool.java:344) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:178) [graylog.jar:?]
	at org.graylog2.bootstrap.Main.main(Main.java:50) [graylog.jar:?]

What you expected to happen:
The StatefulSet provided by the chart already has an initContainer that makes a few adjustments to the journal volume. The chart can be easily patched to include a quick shallow chown.

How to reproduce it (as minimally and precisely as possible):
Install this chart providing a PV that points to a journal directory owned by root.

Anything else we need to know:
Hold my beer. PR coming up.

[juliohm1978]

I'm cross relating this to the Graylog container repo.

I'm not sure what's the best way to resolve this:

1. Make the Graylog container immune to this problem (issue #76 on their side)

2. Implement a workaround in this chart, but that means introducing an implicit complexity into the chart (see PR thread).

[juliohm1978]

This should be fixed by #12983. No longer an issue in my opinion.