Possible bug: ClassCastException

[alexandru]

I have the following exception in a test that tests a component doing Kamon.init() on its own:

[rs-test-akka.actor.internal-dispatcher-4] ERROR 2023-04-10 21:10:33  Logger : An error occurred while trying to apply an advisor: java.lang.ClassCastException: class akka.routing.RoutedActorCell cannot be cast to class kamon.instrumentation.akka.instrumentations.HasActorMonitor (akka.routing.RoutedActorCell and kamon.instrumentation.akka.instrumentations.HasActorMonitor are in unnamed module of loader 'app')
	at kamon.instrumentation.akka.instrumentations.HasActorMonitor$.actorMonitor(ActorInstrumentation.scala:68)
	at kamon.instrumentation.akka.instrumentations.SendMessageAdvice$.onEnter(ActorInstrumentation.scala:109)
	at akka.routing.RoutedActorCell.sendMessage(RoutedActorCell.scala:138)
	at akka.actor.Cell.sendMessage(ActorCell.scala:326)
	at akka.actor.Cell.sendMessage$(ActorCell.scala:325)
	at akka.actor.ActorCell.sendMessage(ActorCell.scala:410)
	at akka.actor.RepointableActorRef.$bang(RepointableActorRef.scala:182)
	at akka.io.SelectionHandler$SelectorBasedManager$$anonfun$workerForCommandHandler$1.applyOrElse(SelectionHandler.scala:118)
	at akka.actor.Actor.aroundReceive(Actor.scala:537)
	at akka.actor.Actor.aroundReceive$(Actor.scala:535)
	at akka.io.SelectionHandler$SelectorBasedManager.aroundReceive(SelectionHandler.scala:101)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579)
	at akka.actor.ActorCell.invoke(ActorCell.scala:547)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
	at akka.dispatch.Mailbox.run(Mailbox.scala:231)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

Not doing Kamon.init() silences this error, so this is probably due to a misconfiguration of some sort.

But, such errors should never happen. This one looks so bad that I'm thinking of disabling the “instrumentation” entirely, and only exposing our counters, spans, etc., which is what we actually want anyway. So, the other problem I'm seeing is that Kamon doesn't support the use-case of disabling instrumentation. Unless I'm configuring it with kamon.enabled = false, which is not the intention.

Any advice?

Versions used here:

• kamon: 2.6.0
• kanela-agent: 1.0.17
• akka: 2.6.20

[msuter-libc]

I experience a similar issue with kamon.enabled = false

Versions used here:

kamon: 2.6.0
kanela-agent: 1.0.17
play 2.8.19 (2.8.8 reproduces same issue)
HikariCP 5.0.1

java.lang.ClassCastException: class com.zaxxer.hikari.pool.HikariProxyPreparedStatement cannot be cast to class kamon.instrumentation.jdbc.HasDatabaseTags (com.zaxxer.hikari.pool.HikariProxyPreparedStatement and kamon.instrumentation.jdbc.HasDatabaseTags are in unnamed module of loader 'app')
        at kamon.instrumentation.jdbc.CreatePreparedStatementAdvice$.exit(StatementInstrumentation.scala:163)
        at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(ProxyConnection.java:327)
        at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(HikariProxyConnection.java)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jdbcdslog.GenericLoggingHandler.invoke(GenericLoggingHandler.java:30)
        at com.sun.proxy.$Proxy90.prepareStatement(Unknown Source)
        at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:149)

 An error occurred while trying to apply an advisor: java.lang.ClassCastException: class akka.routing.RoutedActorCell cannot be cast to class kamon.instrumentation.akka.instrumentations.HasRouterMonitor (akka.routing.RoutedActorCell and kamon.instrumentation.akka.instrumentations.HasRouterMonitor are in unnamed module of loader 'app')
        at kamon.instrumentation.akka.instrumentations.TerminateMethodAdvice$.onEnter(ActorInstrumentation.scala:144)
        at akka.actor.ActorCell.terminate(ActorCell.scala:410)
        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:519)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:535)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:295)
        at akka.dispatch.Mailbox.run(Mailbox.scala:230)
        at kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedForkJoinPool$TimingRunnable.run(ExecutorInstrumentation.scala:698)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) 

[ivantopo]

Hey @alexandru

Most likely what's happening there is that some of the Akka classes are already loaded by the time Kamon.init() gets called, and Kamon can't change already-loaded classes. Then you end up with a half instrumented application that produces these issues. Right as you said, Kamon cannot disable instrumentation once it has been applied (technically speaking, there are a few things that we could rollback but since it is not possible to disable 100% of the instrumentation, we didn't even try to go on that direction).

What we tend to do in tests is:

• Not initialize Kamon at all, since we ensured that all APIs for metrics and tracing will noop as much as possible if Kamon is not enabled
• Call Kamon.initWithoutAttaching(), which is useful if you want to test producing metrics and traces without requiring any of the automatic instrumentation in place
• Add settings to fork the tests with -javaagent:/path/to/kanela.jar, which ensures Kamon can modify classes even if the test code uses Akka before the call to Kamon.init()

I have two questions for you:

• What did you want to do regarding disabling instrumentation? was it about trying to disable it at runtime?
• Do you think that having an equivalent of initWithoutAttaching via configuration would help? So that you can always just have the Kamon.init() call and control the effects of it via config on the tests classpath?

[alexandru]

@ivantopo thanks for your reply.

It's very possible that Kamon.init was called after Akka was initialized, but I'll have to check.

I was not aware of initWithoutAttaching. Will try it out and see what it does.

[ivantopo]

FYI, this came up with one of our customers and we ended up adding the flag to control attaching via configuration here: #1278. Should be released as part of 2.6.1 pretty soon.