New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dedicated partition for Journal - Graylog Server will not start #2348

Closed
jalogisch opened this Issue Jun 13, 2016 · 6 comments

Comments

Projects
None yet
4 participants
@jalogisch
Member

jalogisch commented Jun 13, 2016

Problem description

If you add a new partition to your system to hold the graylog journal or you have a dedicated partition in your design graylog will not start and throw the following error every second:

2016-06-13_10:05:33.68144 2016-06-13 10:05:33,234 ERROR: org.graylog2.shared.journal.KafkaJournal - Unable to start logmanager.
2016-06-13_10:05:33.68171 kafka.common.KafkaException: Failed to acquire lock on file .lock in /opt/graylog/journal_test_899. A Kafka instance in another process or thread is using this directory.
2016-06-13_10:05:33.68204       at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:98) ~[graylog.jar:?]
2016-06-13_10:05:33.68230       at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95) ~[graylog.jar:?]
2016-06-13_10:05:33.68264       at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) ~[graylog.jar:?]
2016-06-13_10:05:33.68289       at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) ~[graylog.jar:?]
2016-06-13_10:05:33.68319       at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) ~[graylog.jar:?]
2016-06-13_10:05:33.68345       at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) ~[graylog.jar:?]
2016-06-13_10:05:33.68374       at scala.collection.TraversableLike$class.map(TraversableLike.scala:245) ~[graylog.jar:?]
2016-06-13_10:05:33.68400       at scala.collection.AbstractTraversable.map(Traversable.scala:104) ~[graylog.jar:?]
2016-06-13_10:05:33.68429       at kafka.log.LogManager.lockLogDirs(LogManager.scala:95) ~[graylog.jar:?]
2016-06-13_10:05:33.68455       at kafka.log.LogManager.<init>(LogManager.scala:57) ~[graylog.jar:?]
2016-06-13_10:05:33.68484       at org.graylog2.shared.journal.KafkaJournal.<init>(KafkaJournal.java:261) [graylog.jar:?]
2016-06-13_10:05:33.68516       at org.graylog2.shared.journal.KafkaJournal$$FastClassByGuice$$8f50433e.newInstance(<generated>) [graylog.jar:?]
2016-06-13_10:05:33.68543       at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) [graylog.jar:?]
2016-06-13_10:05:33.68572       at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) [graylog.jar:?]
2016-06-13_10:05:33.68599       at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105) [graylog.jar:?]
2016-06-13_10:05:33.68627       at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) [graylog.jar:?]

.....

Steps to reproduce the problem

taking OVA 2.0.2 add additional disk to the VM, create partition, format, mount, copy journal

  • cfdisk /dev/sdb
  • mkfs.ext4 /dev/sdb1
  • cd /opt/graylog/
  • mkdir journal_test
  • chown graylog:graylog journal
  • mount /dev/sdb1 /opt/graylog/journal_test
  • graylog-ctl stop graylog-server
  • rsync -avzP --stats /var/opt/graylog/data/journal/* journal_test/
  • vi conf/graylog
  • graylog-ctl start graylog-server
  • tail -f /var/log/graylog/server/current

Environment

  • Graylog Version: 2.0.2 (OVA and any other)

@jalogisch jalogisch changed the title from New Partition for Journal breaks to new Partition for Journal - Graylog Server will not start Jun 13, 2016

@jalogisch jalogisch changed the title from new Partition for Journal - Graylog Server will not start to dedicated partition for Journal - Graylog Server will not start Jun 13, 2016

@jalogisch

This comment has been minimized.

Member

jalogisch commented Jun 13, 2016

you need to create a sub-dir /opt/graylog/journal_test/journal and it will work

@joschi

This comment has been minimized.

Contributor

joschi commented Jun 13, 2016

chown graylog:graylog journal
mount /dev/sdb1 /opt/graylog/journal_test

Just FYI, that's the wrong order. First mount the file system, then run chown. But that's not the reason for Graylog (or rather the Kafka Journal) failing.

I could reproduce the issue with ext4 in the official virtual machine image (https://packages.graylog2.org/releases/graylog-omnibus/ova/graylog-2.0.2-1.ova), but I think it's related to the filesystem itself.

Using XFS as the filesystem for the new journal partition, everything works fine (even without an additional directory below the partition root).

@jalogisch

This comment has been minimized.

Member

jalogisch commented Jun 13, 2016

if you use btrfs this is working also fine - like the lost+found confuse here

@hc4

This comment has been minimized.

Contributor

hc4 commented Jun 15, 2016

Also using btrfs with compression on loop dev - works great (comression ratio 1:5)

@kroepke

This comment has been minimized.

Member

kroepke commented Jun 20, 2016

Kafka is really picky with what's in the directory, this is not something we can fix ourselves (regarding the lost+found).
I would say this needs documentation, because simply adding a subdirectory should do the trick.

@jalogisch

This comment has been minimized.

Member

jalogisch commented Jun 24, 2016

@kroepke i will highlight this in the documentation.

but my advice would be - if possible to check internal if the Journal is broken and display a more useful message!

jalogisch added a commit that referenced this issue Jun 24, 2016

added Information for journal partitions
additional to the needed update in the documentation i had created a note in the graylog.conf about extra partitions for the journal

ref: #2348

jalogisch added a commit to Graylog2/documentation that referenced this issue Jun 24, 2016

dedicated partition for kafka note
added information about dedicated partition for journal, that it can be found inside the docs

re: Graylog2/graylog2-server#2348

@joschi joschi closed this in af3d684 Jun 29, 2016

joschi added a commit that referenced this issue Jun 29, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment