HdrHistogram OutOfBoundsException

[macobo]

This has been reported before: #211, but it has since been closed - I've since encountered the same issue ,multiple times using version 0.5.1.

I have several binaries running all the same code, but some of them rarely stuck in the following error loop (one log line per second or so):

Full stack trace:

2015-11-13 18:11:53 ERROR [ActorSystem(backfillworker)] ActorSystemImpl:66 - Uncaught error from thread [backfillwo
rker-akka.actor.default-dispatcher-108646]
java.lang.ArrayIndexOutOfBoundsException: value outside of histogram covered range. Caused by: java.lang.IndexOutOfBoundsException: index 4620
        at org.HdrHistogram.AbstractHistogram.handleRecordException(AbstractHistogram.java:434)
        at org.HdrHistogram.AbstractHistogram.recordSingleValue(AbstractHistogram.java:426)
        at org.HdrHistogram.AbstractHistogram.recordValue(AbstractHistogram.java:339)
        at kamon.metric.instrument.HdrHistogram.record(Histogram.scala:115)
        at akka.kamon.instrumentation.ActorCellInstrumentation.aroundBehaviourInvoke(ActorCellInstrumentation.scala:69)
        at akka.actor.ActorCell.invoke(ActorCell.scala:483)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
        at akka.dispatch.Mailbox.run(Mailbox.scala:221)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

[htimur]

+1 here, have the same exception, using version 0.5.1

[htimur]

I'm still getting the exception, running kamon 0.5.2

java.lang.ArrayIndexOutOfBoundsException: value outside of histogram covered range. Caused by: java.lang.IndexOutOfBoundsException: index 4632
        at org.HdrHistogram.AbstractHistogram.handleRecordException(AbstractHistogram.java:441) ~[HdrHistogram-2.1.7.jar:2.1.7]
        at org.HdrHistogram.AbstractHistogram.recordSingleValue(AbstractHistogram.java:433) ~[HdrHistogram-2.1.7.jar:2.1.7]
        at org.HdrHistogram.AbstractHistogram.recordValue(AbstractHistogram.java:346) ~[HdrHistogram-2.1.7.jar:2.1.7]
        at kamon.metric.instrument.HdrHistogram.record(Histogram.scala:115) ~[kamon-core_2.11-0.5.2.jar:0.5.2]
        at akka.kamon.instrumentation.ActorCellInstrumentation.aroundBehaviourInvoke(ActorCellInstrumentation.scala:69) ~[kamon-akka_2.11-0.5.2.jar:0.5.2]
        at akka.actor.ActorCell.invoke(ActorCell.scala:483) [akka-actor_2.11-2.3.14.jar:na]
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) [akka-actor_2.11-2.3.14.jar:na]
        at akka.dispatch.Mailbox.run(Mailbox.scala:220) [akka-actor_2.11-2.3.14.jar:na]
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [akka-actor_2.11-2.3.14.jar:na]
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.11.7.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.11.7.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.11.7.jar:na]
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.11.7.jar:na]

[trobert]

This happens to me when timeInMailbox metric get over 1 hour. While this is not a normal condition, I wish Kamon handle this better than crashing the underlying actor with an exception. Under these exceptionnal circumstances, I think it's preferable that Kamon gives slighty wrong metrics rather than altering the behaviour of the monitored application.

Possible solutions could include:

• Provide a way to enable HdrHistogram autoResize feature
• Drop the value or (better IMO) cap it to the highest value

[alno]

Just ran into the same issue with some long-running task traces and after some investigation found that histogram range in Kamon can be configured.

Its default value is in kamon.metric.default-instrument-settings.histogram.highest-trackable-value and equal to 1 hour in nanoseconds, but you can override it specifically for trace elapsed-time metrics by specifying kamon.metric.instrument-settings.trace.elapsed-time.highest-trackable-value as described here.

So, just specify some reasonable value for this setting in your application.conf and the problem should be solved.

[ivantopo]

thanks for sharing that, @alno! I'm closing this one and please take a look at this comment: #232 (comment), that issue is now "tracking" this problem too.

[samikrc]

I am a bit confused about how to change the settings to solve this case. @alno mentions to change kamon.metric.instrument-settings.trace.elapsed-time.highest-trackable-value but the documentation is not very clear. What does the "highest-trackable-value" mean for "elapsed-time"?

I know that this issue is being fixed through code change (335), but wanted to see if this quick fix works.

[Basawaraj-Pagade]

Hello, What is the suggested value for "highest-trackable-value" since it is not clear on the suggested value. Appreciate if you could share more details around the same.