Cluster sharding metrics: Histogram recorded value cannot be negative

[scullxbones]

Started seeing this error immediately in our staging environment after upgrading Kamon from 2.1.8. In the short term I will try turning off the cluster sharding metrics via configuration before this code goes to production.

The only thing I see that could contribute is #917:

• this actor does passivate
• passivations are tracked as of Add KamonRemoteInstrument #917
• the error is occurring in the entities which is treated as termination

Is it possible that terminations are double counted for passivating actors?

Versions:

Kamon: 2.1.11
Akka: 2.5.32 & Akka-Http: 10.2.3
Spray/Play: N/A

Stack trace:

java.lang.ArrayIndexOutOfBoundsException: Histogram recorded value cannot be negative.
    at org.HdrHistogram.ShadedAbstractHistogram.countsArrayIndex(AbstractHistogram.java:2181)
    at org.HdrHistogram.ShadedAbstractHistogram.recordSingleValue(AbstractHistogram.java:478)
    at org.HdrHistogram.ShadedAbstractHistogram.recordValue(AbstractHistogram.java:397)
    at org.HdrHistogram.BaseAtomicHdrHistogram.recordValue(BaseAtomicHdrHistogram.java:41)
    at kamon.metric.Histogram$Atomic.record(Histogram.scala:64)
    at kamon.instrumentation.akka.AkkaClusterShardingMetrics$ShardingInstruments$$anon$1.$anonfun$run$1(AkkaClusterShardingMetrics.scala:92)
    at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:553)
    at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:551)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:920)
    at kamon.instrumentation.akka.AkkaClusterShardingMetrics$ShardingInstruments$$anon$1.run(AkkaClusterShardingMetrics.scala:92)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

Log message:

Failed to record value [-1] on [akka.cluster.sharding.shard.hosted-entities,{type=xxx,system=yyy}] because the value is outside of the configured range. The recorded value was adjusted to the highest trackable value [3600000000000]. You might need to change your dynamic range configuration for this metric

[cipriansofronia]

Still having the issue on 2.1.12

[SimunKaracic]

Yup, this is probably my bad.
Looking into it

[scullxbones]

No worries if it is, was just trying to help get to root cause to make diagnosis and fix easier.

I appreciate the work you and the rest of the team do on Kamon, it's a great product!