IllegalAccessError for net.bytebuddy.dynamic.Nexus.NO_QUEUE

[ahatzz11]

This is maybe a duplicate of #616, but that was a couple years ago and the library we are using is a minimum version of 1.12.0.

We are using log4j-jndi-be-gone to help with some log4j mitigation. They are pointing to a minimum version of 1.12.0 https://github.com/nccgroup/log4j-jndi-be-gone/blob/main/build.gradle#L18

When running our application with the following JAVA_OPTS:

-javaagent:/opt/datadog/dd-java-agent.jar -javaagent:/opt/log4j-jndi-be-gone/log4j-jndi-be-gone.jar '

We get the following in some of our applications:

Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:500)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:512)
Caused by: java.lang.IllegalAccessError: tried to access field net.bytebuddy.dynamic.Nexus.NO_QUEUE from class net.bytebuddy.dynamic.NexusAccessor
	at net.bytebuddy.dynamic.NexusAccessor.<init>(NexusAccessor.java:68)
	at net.bytebuddy.agent.builder.AgentBuilder$InitializationStrategy$SelfInjection$Split.<init>(AgentBuilder.java:3169)
	at net.bytebuddy.agent.builder.AgentBuilder$Default.<init>(AgentBuilder.java:8397)
	at net.bytebuddy.agent.builder.AgentBuilder$Default.<init>(AgentBuilder.java:8385)
	at trust.nccgroup.jndibegone.hooks.JndiLookup__lookup.hook(JndiLookup__lookup.java:51)
	at trust.nccgroup.jndibegone.Agent.load(Agent.java:26)
	at trust.nccgroup.jndibegone.PreMain.premain(PreMain.java:27)
	... 6 more
FATAL ERROR in native method: processing of -javaagent failed
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f6ab2ef0324, pid=1, tid=11
#
# JRE version: OpenJDK Runtime Environment (9.0+12) (build 9.0.4+12-Debian-4)
# Java VM: OpenJDK 64-Bit Server VM (9.0.4+12-Debian-4, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libc.so.6+0x36324]  abort+0x244

Any help would be appreciated!

[raphw]

Did you build the agent with the latest BB? This field is removed, the latest version should no longer expect it.

[raphw]

I think the agent does not shade Byte Buddy into a different package, and there might be an older version of Byte Buddy already on the class path. The Nexus is not used in the agent, but I think this could explain the crash.

[ahatzz11]

@raphw Thanks for the reply, I am honestly not sure on your question so I cross-referenced this issue on their repo to maybe get some more expertise looking at this. Maybe them shading bytebuddy would be the best path forward here. If you have any other suggestions please let me know!

[raphw]

Do you have BB on your class path? Try using the same version there as in the agent, as a backup.

[ahatzz11]

@raphw That is something else I'm not sure on. We tried to add this agent at a global level to our platform and a small % of applications failed, so my guess is that subset of apps probably do have an older version of BB on the classpath.

I took a swing at updating the javagent to shade bytebuddy to avoid this conflict (since updating each of the individual apps would probably be a bummer). I'm not sure if you have any experience in this area, but if you'd be able to take a look that would be sweet: nccgroup/log4j-jndi-be-gone#7 (comment)

[raphw]

Looks good but I'd choose a package in the same hierarchy: trust.nccgroup.jndibegone

[ahatzz11]

That's a good idea, I'll update 👍

[ahatzz11]

@raphw just wanted to follow-up and let you know that shadowing worked. I appreciate all your help!