App fails to start with java.lang.NoClassDefFoundError on 2.19

[RomanLebid]

After uprgading kamon-bundle to 2.19 with kanela-agent 1.0.9, application instantly fails after Kamon.init() call with the following error:

[main] ERROR 2021-05-27 22:57:17  Logger : Unable to start Kanela Agent.
Please remove -javaagent from your startup arguments and contact Kanela support.: java.lang.NoClassDefFoundError: redis/clients/jedis/commands/ProtocolCommand
	at kamon.instrumentation.jedis.JedisInstrumentation.<init>(JedisInstrumentation.scala:11)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at java.base/java.lang.Class.newInstance(Class.java:584)
	at kanela.agent.InstrumentationLoader.lambda$loadInstrumentation$b85f757d$1(InstrumentationLoader.java:58)
	at kanela.agent.libs.io.vavr.control.Try.of(Try.java:75)
	at kanela.agent.InstrumentationLoader.loadInstrumentation(InstrumentationLoader.java:56)
	at kanela.agent.InstrumentationLoader.lambda$null$1(InstrumentationLoader.java:46)
	at kanela.agent.libs.io.vavr.collection.List.flatMap(List.java:902)
	at kanela.agent.InstrumentationLoader.lambda$load$4(InstrumentationLoader.java:46)
	at kanela.agent.libs.io.vavr.collection.List.map(List.java:1042)
	at kanela.agent.InstrumentationLoader.load(InstrumentationLoader.java:43)
	at kanela.agent.Kanela.lambda$null$0(Kanela.java:79)
	at kanela.agent.util.classloader.InstrumentationClassPath.use(InstrumentationClassPath.java:84)
	at kanela.agent.Kanela.lambda$start$1(Kanela.java:68)
	at kanela.agent.util.Execution.timed(Execution.java:32)
	at kanela.agent.util.Execution.runWithTimeSpent(Execution.java:41)
	at kanela.agent.Kanela.start(Kanela.java:67)
	at kanela.agent.Kanela.premain(Kanela.java:47)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)

The only thing that allows application to start correctly is disabling redis instrumentation explicitly via

kanela.modules.redis.instrumentations = []

[dpsoft]

@RomanLebid you have the jedis dependency in your application classpath? Do you have multiple versions?

[SimunKaracic]

Well, my bad. I'll fix this asap

[RomanLebid]

@RomanLebid you have the jedis dependency in your application classpath? Do you have multiple versions?

@dpsoft No, we don't have any redis clients at all (double-checked that with sbt:api> dependencyTree).

It's just a simple Akka HTTP app with a couple of endpoints.
I know that it should be possible to use kamon-akka-http, but for consistency we use the
kamon-bundle + kamon-prometheus combo almost everywhere

[SimunKaracic]

This is my bad, the new version of Kamon will fail to load for everyone who doesn't have jedis on the classpath :/
I did not realize that using an ElementMatcher in InstrumentationBuilder with a class from a library will fail if the class is not on the classpath.

[SimunKaracic]

Fixed with 2.1.20.
Thanks for the patience, it's releasing now, should be available within the hour

[channingwalton]

Hi, we just saw this error in 2.1.21, is it possible there was a regression?

[ivantopo]

Hey @channingwalton, can you please double check the version that ended up in the classpath? I tried a minute ago and couldn't reproduce the issue.

[channingwalton]

Hi @ivantopo, will do.

We went from 2.1.18 to 2.1.21, saw no problems in QA, but typically everything went wrong in prod.

It'll take me a little time to investigate though - probably Monday.

[channingwalton]

I've just seen an error in our QA environment with 2.1.20 too. I'm reverting to 2.1.18 which was working to check that there isn't anything else going on.

[channingwalton]

Hi, FYI our QA was fine with 2.1.18.

[ivantopo]

Do you see the same error with 2.2.0 (released on Friday)?

[channingwalton]

I'll try it now …

[channingwalton]

Yes, we are seeing these exceptions on start up:

An error occurred while trying to apply an advisor: java.lang.NullPointerException

An error occurred while trying to apply an advisor: java.lang.NoClassDefFoundError: redis/clients/jedis/commands/ProtocolCommand

Kamon is initialised on start up, and the reconfigured later, so I don't think its the usual issue:

  def main(args: Array[String]): Unit = {
    Kamon.init()
    ...

and we depend on kamon-influxdb and kamon-bundle.

I am very willing to accept we are doing something else wrong of course.

[ivantopo]

Can you please double-check the runtime classpath? We saw a few times in the past that users had old artifacts in the classpath and those were being picked up before the updated ones.

[channingwalton]

Will do.