Blockhound + Opentelemetry agent fail on install

[friscoMad]

We are using the opentelemtry java agent via JVM params and when Blockhound tries to install it fails as it tries to instrument opentelemetry loaded code.

[Byte Buddy] ERROR java.util.concurrent.FutureTask [null, module java.base, Thread[main,5,main], loaded=true]                                                                                                                                                          │
│ reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Resolution$NoSuchTypeException: Cannot resolve type description for io.opentelemetry.javaagent.bootstrap.VirtualFieldInstalledMarker                                                                             ││     at reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Resolution$Illegal.resolve(TypePool.java:190)                                                                                                                                                             │
│     at reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType.toErasure(TypePool.java:6931)                                                                                                                            ││     at reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType$TokenList.get(TypePool.java:7012)                                                                                                                        │
│     at reactor.blockhound.shaded.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType$TokenList.get(TypePool.java:6958)                                                                                                                        ││     at java.base/java.util.AbstractList$Itr.next(AbstractList.java:371)                                                                                                                                                                                                │
│     at reactor.blockhound.shaded.net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default.doAnalyze(MethodGraph.java:745)                                                                                                                                           ││     at reactor.blockhound.shaded.net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default.compile(MethodGraph.java:668)                                                                                                                                             │
│     at reactor.blockhound.shaded.net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$AbstractBase.compile(MethodGraph.java:519)                                                                                                                                        ││     at reactor.blockhound.shaded.net.bytebuddy.dynamic.scaffold.inline.DecoratingDynamicTypeBuilder.toTypeWriter(DecoratingDynamicTypeBuilder.java:496)                                                                                                                │
│     at reactor.blockhound.shaded.net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase$UsingTypeWriter.make(DynamicType.java:4062)                                                                                                                                    ││     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:12510)                                                                                                                            │
│     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12445)                                                                                                                              ││     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1800(AgentBuilder.java:12154)                                                                                                                            │
│     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12936)                                                                                                           ││     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12866)                                                                                                           │
│     at java.base/java.security.AccessController.doPrivileged(Native Method)                                                                                                                                                                                            ││     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)                                                                                                                                 │
│     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12388)                                                                                                                              ││     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)                                                                                                               │
│     at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)                                                                                                                                                                        ││     at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)                                                                                                                                                                      │
│     at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)                                                                                                                                                                           ││     at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:167)                                                                                                                                                             │
│     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                                                                                                                                  ││     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)                                                                                                                                                                │
│     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                                                                                                                                                        ││     at java.base/java.lang.reflect.Method.invoke(Method.java:566)                                                                                                                                                                                                      │
│     at reactor.blockhound.shaded.net.bytebuddy.utility.Invoker$Dispatcher.invoke(Unknown Source)                                                                                                                                                                       ││     at reactor.blockhound.shaded.net.bytebuddy.utility.dispatcher.JavaDispatcher$Dispatcher$ForNonStaticMethod.invoke(JavaDispatcher.java:1032)                                                                                                                        │
│     at reactor.blockhound.shaded.net.bytebuddy.utility.dispatcher.JavaDispatcher$ProxiedInvocationHandler.invoke(JavaDispatcher.java:1162)                                                                                                                             ││     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.$Proxy86.retransformClasses(Unknown Source)                                                                                                                                                               │
│     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector$ForRetransformation.doApply(AgentBuilder.java:8395)                                                                                                           ││     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector.apply(AgentBuilder.java:8210)                                                                                                                                 │
│     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy.apply(AgentBuilder.java:5907)                                                                                                                                           ││     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:11521)                                                                                                                                                   │
│     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(AgentBuilder.java:11421)                                                                                                                                                   ││     at reactor.blockhound.shaded.net.bytebuddy.agent.builder.AgentBuilder$Default$Delegator.installOn(AgentBuilder.java:13218)                                                                                                                                         │
│     at reactor.blockhound.BlockHound$Builder.instrument(BlockHound.java:597)

Expected Behavior

Open telemetry and blockhound can work together

Actual Behavior

Instrumentation fails

Steps to Reproduce

java -javaagent:/opt/otel/opentelemetry-javaagent-all.jar

@Test
void reproCase() {
   BlockHound.install()
}

Possible Solution

This reported issue in ByteBuddy seems to be related:
raphw/byte-buddy#1320

Your Environment

• Reactor version(s) used:
• Other relevant libraries versions (eg. netty, ...): opentelemetry 1.24 (I will do some test with a recent version in case it was fixed there)
• JVM version (java -version): 11
• OS and version (eg uname -a):

[pderop]

Hi @friscoMad ,

Can you try to run your test like this: instead of calling BlockHound.install() from your test in the java code, declare the blockhound agent using a -javaagent option, like you do with your open telemetry agent.

but make sure you declare the BlockHound agent before the open telemetry agent.

it works for me when I run my sample application like this:

java -javaagent:blockhound-1.0.9.RELEASE.jar -javaagent:opentelemetry-javaagent-2.4.0-SNAPSHOT.jar  -jar target/echo-0.0.1-SNAPSHOT.jar

note that if I declare the blockhound agent after the opentelemetry agent, then I reproduce the same problem you have reported.

I have tested with jdk-8, jdk-11, jdk-17, jdk-21, and jdk-22

do not forget to use -XX:+AllowRedefinitionToAddDeleteMethods in case you use a JDK13+ version.

let me know.

thanks.

[friscoMad]

Thanks, we are trying to avoid adding more agents to the command line, so our solution is to move the OTel agent to attach it on runtime using this contrib lib:
https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/runtime-attach/README.md
So far it appears to be working and both seem to be doing their thing, even when Blockhound is attached after OTel, which is interesting.

[pderop]

ok, I'm closing this issue, reopen it if necessary.
thanks.