Confusing error: Logstash failed to create queue {"exception"=>"Page file size is too small to hold elements" ... #8480

colinsurprenant · 2017-10-11T22:04:48Z

Logstash failed to create queue {"exception"=>"Page file size is too small to hold elements" ...

The error is confusing for end users. We should more clearly state that this is potentially a corruption problem. See #8098

The text was updated successfully, but these errors were encountered:

davtex · 2019-03-29T16:33:42Z

Just happened to me on 6.6.2, had to purge queue to fix the problem.

colinsurprenant · 2019-03-29T16:57:07Z

@davtex - Next time, make sure you backup your queue dir somewhere else where we can run some diagnostics on it. Also, if you could provide more contextual information it would help in understanding what events lead to this problem (any warnings or error logs?, any crash?, etc).

In 6.6.2 there is bin/pqcheck and bin/pqrepair, it would be interesting to get the bin/pqcheck output on the "corrupted" queue.

davtex · 2019-03-29T17:31:37Z

@colinsurprenant Our primary logstash was inoperable since 18th March and I have just noticed today since secondary was able to keep up with load just until today, when I noticed that Logstash is struggling and there is a 3min gap in log processing.

Initial issue was this:

[2019-03-29T20:18:26,286][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
org.logstash.LockException: The queue failed to obtain exclusive access, cause: Lock held by this virtual machine on lock path: /xxx/data/logstash/queue/main/.lock

This was fixed by setting correct permissions on /xxx/data/logstash (logstash:logstash, was root:logstash for some reason).

Next issue:

[2019-03-29T20:19:19,525][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
java.io.IOException: Page file size is too small to hold elements

I checked queue size manually and it was low, so I did not think twice - wverythink was OK afterwards..
[root@XXXlogstash]# du -sm *
1 dead_letter_queue
1 lost+found
5 queue
1 uuid
[root@XXX logstash]# rm -rf queue/*

I have checked monitoring in Kibana and I can now say that the queue was empty. I suspect PQ could have been corrupted for the same reason as our problem with .lock file - change of file wonership mid-flight.

David.

bryantcj52 · 2019-05-20T14:59:14Z

Also happening on 6.7.2, pqcheck/pqrepair did not help. FIle permissions were all correct.

ilmarh · 2019-06-06T17:28:26Z

Logstash 6.7.1 here stuck with the same issue. PQ* utils output:

[root@host01 bin]# ./pqcheck /opt/elk-apps/logstash_k2e_docker_6.7.1/data/queue/common/
Checking queue dir: /opt/elk-apps/logstash_k2e_docker_6.7.1/data/queue/common
checkpoint.24667, fully-acked: NO, page.24667 size: 67108864
pageNum=24667, firstUnackedPageNum=0, firstUnackedSeqNum=536864297, minSeqNum=536860047, elementCount=6086, isFullyAcked=no
checkpoint.head, fully-acked: NO, page.24669 size: 67108864
pageNum=24669, firstUnackedPageNum=24667, firstUnackedSeqNum=0, minSeqNum=0, elementCount=0, isFullyAcked=no
[root@host01 bin]# ./pqrepair /opt/elk-apps/logstash_k2e_docker_6.7.1/data/queue/common/
[root@host01 bin]# ./pqcheck /opt/elk-apps/logstash_k2e_docker_6.7.1/data/queue/common/
Checking queue dir: /opt/elk-apps/logstash_k2e_docker_6.7.1/data/queue/common
checkpoint.head, fully-acked: NO, page.24669 size: 67108864
pageNum=24669, firstUnackedPageNum=24667, firstUnackedSeqNum=0, minSeqNum=0, elementCount=0, isFullyAcked=no
checkpoint.24667, fully-acked: NO, page.24667 size: 67108864
pageNum=24667, firstUnackedPageNum=0, firstUnackedSeqNum=536864297, minSeqNum=536860047, elementCount=6086, isFullyAcked=no
[root@host01 bin]#

Permissions:

[root@host01 bin]# ls -la /opt/elk-apps/logstash_k2e_docker_6.7.1/data/queue/common/
total 131100
drwxr-xr-x. 2 logstash logstash     4096 Jun  6 20:05 .
drwxr-xr-x. 4 logstash logstash     4096 Jun  6 20:05 ..
-rw-r--r--. 1 logstash logstash       34 Jun  6 20:05 checkpoint.24667
-rw-r--r--. 1 logstash logstash       34 Jun  6 20:05 checkpoint.head
-rw-r--r--. 1 logstash logstash        0 Jun  6 20:05 .lock
-rw-r--r--. 1 logstash logstash 67108864 Jun  6 20:05 page.24667
-rw-r--r--. 1 logstash logstash 67108864 Jun  6 20:05 page.24669
-rw-r--r--. 1 logstash logstash        4 Jun  6 20:05 .queue-version

No space problems detected (50Gb free on root, that holds /opt):

[root@host01 bin]# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/vg_system-lv_root      55G  3.0G   50G   6% /
devtmpfs                           16G     0   16G   0% /dev
tmpfs                              16G     0   16G   0% /dev/shm
tmpfs                              16G  818M   15G   6% /run
tmpfs                              16G     0   16G   0% /sys/fs/cgroup
/dev/sda1                         2.0G  247M  1.6G  14% /boot
/dev/mapper/vg_system-lv_var      1.5T  551G  901G  38% /var
/dev/mapper/vg_system-lv_var_log   19G  3.6G   14G  21% /var/log
/dev/mapper/vg_system-lv_tmp      9.1G  354M  8.3G   5% /tmp
/dev/mapper/vg_system-lv_home     5.4G   30M  5.1G   1% /home
overlay                           1.5T  551G  901G  38% /var/lib/docker/overlay2/ed59fa8fdd5419af843f8699e295ec42c19d7fef345fa789c557ce1a89631d7a/merged
overlay                           1.5T  551G  901G  38% /var/lib/docker/overlay2/69ac96ee8f8ace037f60fd2fad0483ca348e8cff0ac60f55f4015b6ba55825d8/merged
tmpfs                             3.2G     0  3.2G   0% /run/user/465650168
tmpfs                             3.2G     0  3.2G   0% /run/user/465628218
shm                                64M     0   64M   0% /var/lib/docker/containers/eb6e6892a8eb2708bbd90f51526cfec569a2bb29da978d5d1691f13011a065cf/mounts/shm
overlay                           1.5T  551G  901G  38% /var/lib/docker/overlay2/d81446baf5054ec7b488107da8020e909c11f5ee67238a7533a9038b4ee37d24/merged
shm                                64M     0   64M   0% /var/lib/docker/containers/6770cae4bd05e8584921a42c9a6ca4474833f3a200522322d0bcc1b05b2f471e/mounts/shm
shm                                64M     0   64M   0% /var/lib/docker/containers/c9746d8c700a6826c9fb83dc241df289d18b80e8b983a199936c5f7e1d399d82/mounts/shm

Have to delete files from dir in order to make Logstash work again.
Is there any way to fix problem with out loosing persistant data?

nicenemo · 2019-11-11T12:18:45Z

Also happens on 7.4.2

colinsurprenant · 2019-11-15T16:48:42Z

@nicenemo could you please provide more information, like the exact error/exception log, the content of the queue dir and the output of bin/pqcheck. Also, all contextual information to what might have lead to this; did logstash crash? was it restarted? was there any errors or warnings? etc.

nicenemo · 2019-11-18T07:53:10Z

Since I had a backup of all data as JSON files, I restarted after clearing the queues. There was nothing meaningful in the logs. Op vr 15 nov. 2019 17:49 schreef Colin Surprenant <notifications@github.com

…

: @nicenemo <https://github.com/nicenemo> could you please provide more information, like the exact error/exception log, the content of the queue dir and the output of bin/pqcheck. Also, all contextual information to what might have lead to this; did logstash crash? was it restarted? was there any errors or warnings? etc. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8480?email_source=notifications&email_token=AAUNS5E3CFAN3QJHGWRCB23QT3HJHA5CNFSM4D6YYQI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEGA32Q#issuecomment-554438122>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAUNS5DO6LUNXTMI4R7EJSTQT3HJHANCNFSM4D6YYQIQ> .

burghategaurao · 2019-12-10T06:18:22Z

@colinsurprenant
Logstash is getting stopped and then if we restart logstash then we face Page Size Too Small issue.
PFB stacktrace.
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console url template {:scheme=>nil, :user=>nil, :password=>nil, :host=>"URLTEMPLATE", :port=>443, :path=>nil} url template {:scheme=>nil, :user=>nil, :password=>nil, :host=>"URLTEMPLATE", :port=>443, :path=>nil} [ERROR] 2019-12-10 06:07:18.651 [Converge PipelineAction::Create<main>] AbstractPipelineExt - Logstash failed to create queue. java.io.IOException: Page file size is too small to hold elements at org.logstash.ackedqueue.io.MmapPageIOV2.mapFile(MmapPageIOV2.java:275) ~[logstash-core.jar:?] at org.logstash.ackedqueue.io.MmapPageIOV2.recover(MmapPageIOV2.java:134) ~[logstash-core.jar:?] at org.logstash.ackedqueue.Queue.open(Queue.java:210) ~[logstash-core.jar:?] at org.logstash.ackedqueue.ext.JRubyAckedQueueExt.open(JRubyAckedQueueExt.java:97) ~[logstash-core.jar:?] at org.logstash.ackedqueue.ext.JRubyWrappedAckedQueueExt.initialize(JRubyWrappedAckedQueueExt.java:39) ~[logstash-core.jar:?] at org.logstash.ackedqueue.QueueFactoryExt.create(QueueFactoryExt.java:37) ~[logstash-core.jar:?] at org.logstash.execution.AbstractPipelineExt.openQueue(AbstractPipelineExt.java:167) [logstash-core.jar:?] at org.logstash.execution.AbstractPipelineExt$INVOKER$i$0$0$openQueue.call(AbstractPipelineExt$INVOKER$i$0$0$openQueue.gen) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:339) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83) [jruby-complete-9.1.13.0.jar:?] at org.jruby.RubyClass.newInstance(RubyClass.java:1022) [jruby-complete-9.1.13.0.jar:?] at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.Block.call(Block.java:124) [jruby-complete-9.1.13.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:289) [jruby-complete-9.1.13.0.jar:?] at org.jruby.RubyProc.call19(RubyProc.java:273) [jruby-complete-9.1.13.0.jar:?] at org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:196) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:192) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:129) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:339) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:84) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.Block.yieldSpecific(Block.java:134) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ext.thread.Mutex.synchronize(Mutex.java:148) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:89) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:193) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:323) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.Block.call(Block.java:124) [jruby-complete-9.1.13.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:289) [jruby-complete-9.1.13.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:246) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104) [jruby-complete-9.1.13.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191] [ERROR] 2019-12-10 06:07:18.666 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"java.io.IOException: Page file size is too small to hold elements", :backtrace=>["org.logstash.execution.AbstractPipelineExt.openQueue(AbstractPipelineExt.java:170)", "or [FATAL] 2019-12-10 06:07:18.746 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateExceptionforPipelineAction::Create

>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:103:in create'", "org/logstash/execution/ConvergeResultExt.java:34:in

add'", "/usr/share/logstash/logstash-core/lib/lo [ERROR] 2019-12-10 06:07:18.755 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

borissnd · 2020-03-19T15:56:48Z

same here. using Logstash 7.4.2. This keeps happening quite frequently.
running pqrepair on the main directory seems to fix the issue temporarily.
Here is the stacktrace:

[2020-03-19T15:36:53,475][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.4.2"}
[2020-03-19T15:37:26,858][INFO ][org.reflections.Reflections] Reflections took 73 ms to scan 1 urls, producing 20 keys and 40 values 
[2020-03-19T15:37:45,629][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
java.io.IOException: Page file size is too small to hold elements
	at org.logstash.ackedqueue.io.MmapPageIOV2.mapFile(MmapPageIOV2.java:275) ~[logstash-core.jar:?]
	at org.logstash.ackedqueue.io.MmapPageIOV2.open(MmapPageIOV2.java:64) ~[logstash-core.jar:?]
	at org.logstash.ackedqueue.Queue.open(Queue.java:193) ~[logstash-core.jar:?]
	at org.logstash.ackedqueue.ext.JRubyAckedQueueExt.open(JRubyAckedQueueExt.java:101) ~[logstash-core.jar:?]
	at org.logstash.ackedqueue.ext.JRubyWrappedAckedQueueExt.initialize(JRubyWrappedAckedQueueExt.java:42) ~[logstash-core.jar:?]
	at org.logstash.ackedqueue.QueueFactoryExt.create(QueueFactoryExt.java:39) ~[logstash-core.jar:?]
	at org.logstash.execution.AbstractPipelineExt.openQueue(AbstractPipelineExt.java:171) [logstash-core.jar:?]
	at org.logstash.execution.AbstractPipelineExt$INVOKER$i$0$0$openQueue.call(AbstractPipelineExt$INVOKER$i$0$0$openQueue.gen) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:831) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183) [jruby-complete-9.2.8.0.jar:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.RubyClass.newInstance(RubyClass.java:915) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183) [jruby-complete-9.2.8.0.jar:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36) [jruby-complete-9.2.8.0.jar:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183) [jruby-complete-9.2.8.0.jar:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.runtime.Block.call(Block.java:129) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.RubyProc.call(RubyProc.java:295) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.RubyProc.call(RubyProc.java:274) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.RubyProc.call(RubyProc.java:270) [jruby-complete-9.2.8.0.jar:?]
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.8.0.jar:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2020-03-19T15:37:45,639][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"java.io.IOException: Page file size is too small to hold elements", :backtrace=>["org.logstash.execution.AbstractPipelineExt.openQueue(AbstractPipelineExt.java:174)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$0$0$openQueue.call(AbstractPipelineExt$INVOKER$i$0$0$openQueue.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:831)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.Block.call(Block.java:129)", "org.jruby.RubyProc.call(RubyProc.java:295)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.RubyProc.call(RubyProc.java:270)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
          create at org/logstash/execution/ConvergeResultExt.java:109
             add at org/logstash/execution/ConvergeResultExt.java:37
  converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:339
[2020-03-19T15:37:45,651][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
[2020-03-19T15:37:45,685][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
[2020-03-19T15:37:45,767][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

yizhang13 · 2020-04-26T17:54:08Z

logstash version 7.6.2, using persistence queue, is this resovled?
what shoud I do to work around?

[2020-04-26T17:44:41,554][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.6.2"}
[2020-04-26T17:44:49,395][INFO ][org.reflections.Reflections] Reflections took 109 ms to scan 1 urls, producing 20 keys and 40 values
[2020-04-26T17:44:53,878][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
java.io.IOException: Page file size is too small to hold elements
        at org.logstash.ackedqueue.io.MmapPageIOV2.mapFile(MmapPageIOV2.java:275) ~[logstash-core.jar:?]
        at org.logstash.ackedqueue.io.MmapPageIOV2.open(MmapPageIOV2.java:64) ~[logstash-core.jar:?]
        at org.logstash.ackedqueue.Queue.open(Queue.java:193) ~[logstash-core.jar:?]
        at org.logstash.ackedqueue.ext.JRubyAckedQueueExt.open(JRubyAckedQueueExt.java:100) ~[logstash-core.jar:?]
        at org.logstash.ackedqueue.ext.JRubyWrappedAckedQueueExt.initialize(JRubyWrappedAckedQueueExt.java:42) ~[logstash-core.jar:?]
        at org.logstash.ackedqueue.QueueFactoryExt.create(QueueFactoryExt.java:39) ~[logstash-core.jar:?]
        at org.logstash.execution.AbstractPipelineExt.openQueue(AbstractPipelineExt.java:176) [logstash-core.jar:?]
        at org.logstash.execution.AbstractPipelineExt$INVOKER$i$0$0$openQueue.call(AbstractPipelineExt$INVOKER$i$0$0$openQueue.gen) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:354) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:143) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:345) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:84) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.RubyClass.newInstance(RubyClass.java:915) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:84) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:540) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183) [jruby-complete-9.2.9.0.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.runtime.Block.call(Block.java:125) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.RubyProc.call(RubyProc.java:274) [jruby-complete-9.2.9.0.jar:?]
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.9.0.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

cskowronnek · 2020-05-16T13:30:03Z

Same here on 7.6.2

mtudisco49 · 2020-06-08T20:02:07Z

same thing happened to me on logstash 7.5
I have several pipelines configured, one main pipeline that receives the information and does some pre-procesing, then it send to differente pipelines for further processing. All this pipelines end up sending information to two pipelines "es" (writes to elasticsearch) and "fs" (writes to filesystem).

Once the filesystem where the FS pipeline writes got full, logstash started reporting errors, when i noticed we made some space on the filesystem, but it never started again. I had to remove the queues and then it was able to start. The filesystem that was full was where FS pipeline writes, not where LS has the queues. I'm also using persitent queues,

Before deleting the queues I run pqcheck and pqrepair but didnt work. pqcheck reported not fully ack so i guess i have lost some information in the queues.

Is this a bug? is there any clue when theres going to be solved?

thanks

ADHB · 2020-08-13T11:16:02Z

Hi,
I use 7.7.0 and I still get corrupted queue in production everytime my container run out of heap memory or disk space. I found a workaround for solving the issue by doing the following:

make a new backup directory
copy the content of the queue directory into the new backup directory
check how big the queue is with: du -h -c queue_backup
delete the content of the original queue directory
docker restart logstash
docker cp queue_backup container:/usr/share/logstash/data/
docker exec -u root -it container chown -R logstash:root /usr/share/logstash/data/queue
docker exec -u root -it container chmod -R 2755 /usr/share/logstash/data/queue

Tests:

docker restart logstash

This will confirm that the queue is not corrupted anymore and logstash have permissions to the new queue folder.

systemctl stop nginx & du -h -c /var/tmp/logstash/queue

Empty the queue. Stop all data input in the node. Wait for 15 minuttes and check the size of the queue. It should be less that the queue_backup (pkt 2). Restart the container again when you feel the queue is stuck.

Check input metrics in Kibana - If you have it set up

aberhallo · 2021-06-29T09:04:34Z

Hi,
we are still facing this issue - with 7.13.2. Deleting the queues solves this, but in production this is not an acceptable solution.

businessbean · 2021-11-30T16:22:26Z

Still happening in 7.14.2 @richardgilm

kaisecheng · 2022-02-24T16:26:12Z

The issue is fixed in v7.17.1

colinsurprenant added the enhancement label Oct 11, 2017

colinsurprenant mentioned this issue Oct 11, 2017

[Bug] [Logstash] Invalid error message when there is a corruption in the path.data and path.queue folders #8098

Closed

henrikno mentioned this issue May 7, 2019

Persistent queue corruption in 6.2.4 and 6.3.2 #9961

Open

colinsurprenant mentioned this issue Dec 17, 2019

[META] 8.x PQ improvements #11447

Open

32 tasks

roaksoax assigned kaisecheng Feb 1, 2022

kaisecheng mentioned this issue Feb 1, 2022

handle fully acked 0 byte PQ pages #13692

Merged

5 tasks

kaisecheng added the status:work-in-progress label Feb 1, 2022

kares added the persistent queues label Feb 21, 2022

kaisecheng closed this as completed Feb 24, 2022

roaksoax removed the status:work-in-progress label Feb 24, 2022

tgueldner-mms mentioned this issue Jun 16, 2022

newest Logstash versions opensearch-project/logstash-output-opensearch#149

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Confusing error: Logstash failed to create queue {"exception"=>"Page file size is too small to hold elements" ... #8480

Confusing error: Logstash failed to create queue {"exception"=>"Page file size is too small to hold elements" ... #8480

colinsurprenant commented Oct 11, 2017

davtex commented Mar 29, 2019

colinsurprenant commented Mar 29, 2019

davtex commented Mar 29, 2019

bryantcj52 commented May 20, 2019

ilmarh commented Jun 6, 2019 •

edited

nicenemo commented Nov 11, 2019

colinsurprenant commented Nov 15, 2019

nicenemo commented Nov 18, 2019 via email

burghategaurao commented Dec 10, 2019 •

edited

borissnd commented Mar 19, 2020

yizhang13 commented Apr 26, 2020

cskowronnek commented May 16, 2020

mtudisco49 commented Jun 8, 2020

ADHB commented Aug 13, 2020

aberhallo commented Jun 29, 2021

businessbean commented Nov 30, 2021

kaisecheng commented Feb 24, 2022

Confusing error: Logstash failed to create queue {"exception"=>"Page file size is too small to hold elements" ... #8480

Confusing error: Logstash failed to create queue {"exception"=>"Page file size is too small to hold elements" ... #8480

Comments

colinsurprenant commented Oct 11, 2017

davtex commented Mar 29, 2019

colinsurprenant commented Mar 29, 2019

davtex commented Mar 29, 2019

bryantcj52 commented May 20, 2019

ilmarh commented Jun 6, 2019 • edited

nicenemo commented Nov 11, 2019

colinsurprenant commented Nov 15, 2019

nicenemo commented Nov 18, 2019 via email

burghategaurao commented Dec 10, 2019 • edited

borissnd commented Mar 19, 2020

yizhang13 commented Apr 26, 2020

cskowronnek commented May 16, 2020

mtudisco49 commented Jun 8, 2020

ADHB commented Aug 13, 2020

aberhallo commented Jun 29, 2021

businessbean commented Nov 30, 2021

kaisecheng commented Feb 24, 2022

ilmarh commented Jun 6, 2019 •

edited

burghategaurao commented Dec 10, 2019 •

edited