dedicated partition for Journal - Graylog Server will not start

[jalogisch]

Problem description

If you add a new partition to your system to hold the graylog journal or you have a dedicated partition in your design graylog will not start and throw the following error every second:

2016-06-13_10:05:33.68144 2016-06-13 10:05:33,234 ERROR: org.graylog2.shared.journal.KafkaJournal - Unable to start logmanager.
2016-06-13_10:05:33.68171 kafka.common.KafkaException: Failed to acquire lock on file .lock in /opt/graylog/journal_test_899. A Kafka instance in another process or thread is using this directory.
2016-06-13_10:05:33.68204       at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:98) ~[graylog.jar:?]
2016-06-13_10:05:33.68230       at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95) ~[graylog.jar:?]
2016-06-13_10:05:33.68264       at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) ~[graylog.jar:?]
2016-06-13_10:05:33.68289       at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) ~[graylog.jar:?]
2016-06-13_10:05:33.68319       at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) ~[graylog.jar:?]
2016-06-13_10:05:33.68345       at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) ~[graylog.jar:?]
2016-06-13_10:05:33.68374       at scala.collection.TraversableLike$class.map(TraversableLike.scala:245) ~[graylog.jar:?]
2016-06-13_10:05:33.68400       at scala.collection.AbstractTraversable.map(Traversable.scala:104) ~[graylog.jar:?]
2016-06-13_10:05:33.68429       at kafka.log.LogManager.lockLogDirs(LogManager.scala:95) ~[graylog.jar:?]
2016-06-13_10:05:33.68455       at kafka.log.LogManager.<init>(LogManager.scala:57) ~[graylog.jar:?]
2016-06-13_10:05:33.68484       at org.graylog2.shared.journal.KafkaJournal.<init>(KafkaJournal.java:261) [graylog.jar:?]
2016-06-13_10:05:33.68516       at org.graylog2.shared.journal.KafkaJournal$$FastClassByGuice$$8f50433e.newInstance(<generated>) [graylog.jar:?]
2016-06-13_10:05:33.68543       at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) [graylog.jar:?]
2016-06-13_10:05:33.68572       at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) [graylog.jar:?]
2016-06-13_10:05:33.68599       at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105) [graylog.jar:?]
2016-06-13_10:05:33.68627       at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) [graylog.jar:?]

.....

Steps to reproduce the problem

taking OVA 2.0.2 add additional disk to the VM, create partition, format, mount, copy journal

• cfdisk /dev/sdb
• mkfs.ext4 /dev/sdb1
• cd /opt/graylog/
• mkdir journal_test
• chown graylog:graylog journal
• mount /dev/sdb1 /opt/graylog/journal_test
• graylog-ctl stop graylog-server
• rsync -avzP --stats /var/opt/graylog/data/journal/* journal_test/
• vi conf/graylog
• graylog-ctl start graylog-server
• tail -f /var/log/graylog/server/current

Environment

• Graylog Version: 2.0.2 (OVA and any other)

[jalogisch]

you need to create a sub-dir /opt/graylog/journal_test/journal and it will work

[joschi]

chown graylog:graylog journal
mount /dev/sdb1 /opt/graylog/journal_test

Just FYI, that's the wrong order. First mount the file system, then run chown. But that's not the reason for Graylog (or rather the Kafka Journal) failing.

I could reproduce the issue with ext4 in the official virtual machine image (https://packages.graylog2.org/releases/graylog-omnibus/ova/graylog-2.0.2-1.ova), but I think it's related to the filesystem itself.

Using XFS as the filesystem for the new journal partition, everything works fine (even without an additional directory below the partition root).

[jalogisch]

if you use btrfs this is working also fine - like the lost+found confuse here

[hc4]

Also using btrfs with compression on loop dev - works great (comression ratio 1:5)

[kroepke]

Kafka is really picky with what's in the directory, this is not something we can fix ourselves (regarding the lost+found).
I would say this needs documentation, because simply adding a subdirectory should do the trick.

[jalogisch]

@kroepke i will highlight this in the documentation.

but my advice would be - if possible to check internal if the Journal is broken and display a more useful message!

[Rostand76]

you can resolve this problem
systemctl stop graylog-server
rm -rf /var/lib/graylog-server/journal
mkdir /var/lib/graylog-server/journal
chown graylog:graylog /var/lib/graylog-server/journal
systemctl restart graylog-server