Skip to content

Ignore appdynamics/cisco multi tenant agent module #326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 25, 2021

Conversation

ryandens
Copy link
Member

Description

When running the Hypertrace Javaagent with the Appdynamics javaagent on 9+ JVM, the following error occurs after ~3 minutes of leaving an app server running with no HTTP traffic

[Cisco-Multi-Tenant-Agent-Bootstrapping] ERROR io.opentelemetry.javaagent.tooling.HelperInjector - Error preparing helpers while processing class okhttp3.OkHttpClient for okhttp. Failed to inject helper classes into instance MultiTenantAgentClassLoader
java.lang.IllegalStateException: Error invoking (accessor)::defineClass
	at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$UsingUnsafeInjection.defineClass(ClassInjector.java:999)
	at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection.injectRaw(ClassInjector.java:251)
	at io.opentelemetry.javaagent.tooling.HelperInjector.injectClassLoader(HelperInjector.java:205)
	at io.opentelemetry.javaagent.tooling.HelperInjector.transform(HelperInjector.java:137)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:10364)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10302)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1600(AgentBuilder.java:10068)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10761)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10699)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10258)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
	at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
	at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
	at java.base/java.lang.Class.getDeclaredMethods(Class.java:2309)
	at MultiTenantAgentClassLoader/org.picocontainer.injectors.AdaptingInjection$1.run(AdaptingInjection.java:203)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at MultiTenantAgentClassLoader/org.picocontainer.injectors.AdaptingInjection.injectionMethodAnnotated(AdaptingInjection.java:200)
	at MultiTenantAgentClassLoader/org.picocontainer.injectors.AdaptingInjection.methodAnnotatedInjectionAdapter(AdaptingInjection.java:171)
	at MultiTenantAgentClassLoader/org.picocontainer.injectors.AdaptingInjection.createComponentAdapter(AdaptingInjection.java:70)
	at MultiTenantAgentClassLoader/org.picocontainer.behaviors.AbstractBehaviorFactory.createComponentAdapter(AbstractBehaviorFactory.java:44)
	at MultiTenantAgentClassLoader/org.picocontainer.behaviors.Caching.createComponentAdapter(Caching.java:46)
	at MultiTenantAgentClassLoader/org.picocontainer.behaviors.AdaptingBehavior.createComponentAdapter(AdaptingBehavior.java:58)
	at MultiTenantAgentClassLoader/org.picocontainer.DefaultPicoContainer.addComponent(DefaultPicoContainer.java:536)
	at MultiTenantAgentClassLoader/org.picocontainer.DefaultPicoContainer.addComponent(DefaultPicoContainer.java:501)
	at MultiTenantAgentClassLoader/org.picocontainer.DefaultPicoContainer.access$400(DefaultPicoContainer.java:84)
	at MultiTenantAgentClassLoader/org.picocontainer.DefaultPicoContainer$AsPropertiesPicoContainer.addComponent(DefaultPicoContainer.java:1157)
	at MultiTenantAgentClassLoader/com.cisco.mtagent.core.AgentPicoContainer.createSingleton(AgentPicoContainer.java:135)
	at MultiTenantAgentClassLoader/com.cisco.mtagent.core.AgentPicoContainer.initializeMultiTenantAgent(AgentPicoContainer.java:108)
	at MultiTenantAgentClassLoader/com.cisco.mtagent.core.AgentPicoContainer.bootstrapMultiAgentInstances(AgentPicoContainer.java:71)
	at MultiTenantAgentClassLoader/com.cisco.mtagent.core.AgentPicoContainer.bootstrapMultiTenantAgent(AgentPicoContainer.java:58)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.cisco.mtagent.entry.MTAgent$2.run(MTAgent.java:324)
Caused by: java.lang.IllegalAccessError: superinterface check failed: class io.opentelemetry.javaagent.instrumentation.okhttp.v3_0.TracingInterceptor (in unnamed module @0x7aeb0422) cannot access class okhttp3.Interceptor (in module MultiTenantAgentClassLoader) because module MultiTenantAgentClassLoader does not export okhttp3 to unnamed module @0x7aeb0422
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
	at java.base/java.lang.ClassLoader$ByteBuddyAccessor$GCVZL2mm.defineClass(Unknown Source)
	at java.base/jdk.internal.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$UsingUnsafeInjection.defineClass(ClassInjector.java:995)
	... 47 more

Testing

I verified this locally using Tomcat 9 on Correto 11 with Hypertrace and Appdynamics installed.

Checklist:

  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works (N/A, we don't have a good way to test other agenets)
  • Any dependent changes have been merged and published in downstream modules

Documentation

Make sure that you have documented corresponding changes in this repository or hypertrace docs repo if required.

@pavolloffay pavolloffay changed the title 🐛 fix conflict with appdynamics/cisco multi tenant agent module Ignore appdynamics/cisco multi tenant agent module Jun 25, 2021
@@ -55,6 +81,16 @@ public Result classloader(ClassLoader classLoader) {
if (name.startsWith("com.singularity.") || name.startsWith("com.yourkit.")) {
return Result.IGNORE;
}
if (CLASS_LOADER_GET_NAME != null) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why testing against name does not work?

What is the name when the getName returns MultiTenantAgentClassLoader

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the log messages, I tried using the name, but I thought that the class loader wasn't namespaced in a cisco package so my first pass at this solution was adding the check "MultiTenantAgentClassLoader".equals(name) to the if(..) condition above. I was surprised when it didn't work, so I went with this one. Turns out, the class name is namedspaced properly and is com.cisco.mtagent.entry.MTAgent$MultiTenantAgentClassLoader so we definitely don't need all of this extra complexity. I'm very happy to simply check for if the class loader class name starts with com.cisco. and being more comortable knowing this should be relatively stable over time 😄 Thanks for asking, much happier with the solution we landed on as a result!

Copy link
Member

@pavolloffay pavolloffay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again final keyword in the places where it does not make any sense.

I would like to understand why you are not using the existing variable name that has the class name. Also the fact that getName is java 9+ makes this more fragile and there should be a way to do it regardless the java version.

@pavolloffay
Copy link
Member

Circling back to using the final keyword for all local vars and parameters #321 (comment). Have you seen this style anywhere in high-quality code bases e.g. Spring, Rxjava, Kafka, JBoss, Google projects? I personally haven't, final is useful in places where it makes sense: constants, not allowing inheritance, very large methods. Otherwise it's just noise and pollutes the code.

@ryandens
Copy link
Member Author

Re: the final keyword, I think I explained my take on why I think it's useful and not noise/pollution when I read code in the other PR (last comment here). I'm not actively invovled in other open source projects, so I can't speak to those, but I've seen it used in a couple closed source projects I've worked on and enforced with the FinalLocalVariableCheck checkstyle rule. Ultimately, IMO, code style is a compromise between everyone that maintains/contributes to a project, so as a I stated on the other PR I'm happy to table the discussion for another time and remove the final keyword from local variables until such a time that we can add automation to enforce whatever decision we come to. As the Hypertrace user base grows, so will our contributor base and having automation to enforce simple code style will help others more easily contribute to the project

if (name.startsWith("com.singularity.") || name.startsWith("com.yourkit.")) {
if (name.startsWith("com.singularity.")
|| name.startsWith("com.yourkit.")
|| name.startsWith("com.cisco.")) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

com.cisco is too restrictive. Imagine someone from cisco using this agent. Look at the upstream class how it excluded new relic classes - the same story they wanted to use the agent internally so the agent had to exclude only NR agent classes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! I'll update this to be com.cisco.mtagent. to match the com.cisco.mtagent.entry.MTAgent$MultiTenantAgentClassLoader

@pavolloffay
Copy link
Member

I'm not actively invovled in other open source projects, so I can't speak to those,

I encourage anybody to have a look, it's a great learning experience on how to write quality code.

@pavolloffay
Copy link
Member

As the Hypertrace user base grows, so will our contributor base and having automation to enforce simple code style will help others more easily contribute to the project

I haven't seen a code style check that would do anything with final. This keyword should be used by the developer's careful judgement when needed.

@ryandens ryandens force-pushed the appdynamics-classloader branch from 941ed55 to 9c1254d Compare June 25, 2021 16:25
@pavolloffay
Copy link
Member

cisco is going to be the top company in classloader exclusions (appd, singularity, now cisco).

@davexroth
Copy link
Contributor

cisco is going to be the top company in classloader exclusions (appd, singularity, now cisco).

This is a good point - @ryandens can you scope the package down?

@ryandens
Copy link
Member Author

cisco is going to be the top company in classloader exclusions (appd, singularity, now cisco).

This is a good point - @ryandens can you scope the package down?

Yep, this is already done! I scoped it down to com.cisco.mtagent which Pavol and I agree on as the appropriate scope

@ryandens ryandens merged commit 0a62233 into main Jun 25, 2021
@ryandens ryandens deleted the appdynamics-classloader branch June 25, 2021 18:02
@ryandens ryandens restored the appdynamics-classloader branch June 25, 2021 18:02
@ryandens ryandens deleted the appdynamics-classloader branch June 25, 2021 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants