Issues attaching to Solaris Hotspot on Solaris 10 and 11.

[FelixMarxIBM]

Hello Rafael,

Not sure if this is a problem on our test environments but we noticed some problems while attaching to Hotspot based JVMs on Solaris 10 and Solaris 11. We noticed that the dorr_call function fails on both environments with slightly different errors.

Solaris 10:

Attaching using net.bytebuddy.agent.VirtualMachine$ForHotSpot
Attach using net.bytebuddy.agent.VirtualMachine$ForHotSpot
Attaching to 16597
Loading /tmp/.instana/javaagent-loader-1.3.61.jar with bootjar=/com.instana.agent.artifact/com.instana/instrumentation-boot/1.1.29/jar,sharedjar=/com.instana.agent.artifact/com.instana/instrumentation-shared/1.0.141/jar,pid=16597,inMemoryClassLoader=true,agentSocket=true
java.lang.UnsatisfiedLinkError: Error looking up function 'door_call': ld.so.1: java: fatal: door_call: can't find symbol
        at com.sun.jna.Function.<init>(Function.java:252)
        at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:604)
        at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:580)
        at com.sun.jna.NativeLibrary.getFunction(NativeLibrary.java:566)
        at com.sun.jna.Library$Handler.invoke(Library.java:243)
        at net.bytebuddy.agent.$Proxy0.door_call(Unknown Source)
        at net.bytebuddy.agent.VirtualMachine$ForHotSpot$Connection$ForJnaSolarisDoor.execute(VirtualMachine.java:1411)
        at net.bytebuddy.agent.VirtualMachine$ForHotSpot.load(VirtualMachine.java:363)
        at net.bytebuddy.agent.VirtualMachine$ForHotSpot.loadAgent(VirtualMachine.java:337)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.instana.agent.loader.AgentLoaderAttach.attach(AgentLoaderAttach.java:424)
        at com.instana.agent.loader.AgentLoaderAttach.run(AgentLoaderAttach.java:150)
        at com.instana.agent.loader.AgentLoaderAttach.parseArgsAndRun(AgentLoaderAttach.java:100)
        at com.instana.agent.loader.AgentLoaderAttach.main(AgentLoaderAttach.java:83)

Solaris 11:

Attaching using net.bytebuddy.agent.VirtualMachine$ForHotSpot
Attach using net.bytebuddy.agent.VirtualMachine$ForHotSpot
Attaching to 11121
Loading /tmp/.instana/javaagent-loader-1.3.61.jar with bootjar=/com.instana.agent.artifact/com.instana/instrumentation-boot/1.1.29/jar,sharedjar=/com.instana.agent.artifact/com.instana/instrumentation-shared/1.0.141/jar,pid=11121,inMemoryClassLoader=true,agentSocket=true
com.sun.jna.LastErrorException: [7] Arg list too long
        at com.sun.jna.Native.invokeInt(Native Method)
        at com.sun.jna.Function.invoke(Function.java:426)
        at com.sun.jna.Function.invoke(Function.java:361)
        at com.sun.jna.Library$Handler.invoke(Library.java:265)
        at net.bytebuddy.agent.$Proxy0.door_call(Unknown Source)
        at net.bytebuddy.agent.VirtualMachine$ForHotSpot$Connection$ForJnaSolarisDoor.execute(VirtualMachine.java:1411)
        at net.bytebuddy.agent.VirtualMachine$ForHotSpot.load(VirtualMachine.java:363)
        at net.bytebuddy.agent.VirtualMachine$ForHotSpot.loadAgent(VirtualMachine.java:337)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.instana.agent.loader.AgentLoaderAttach.attach(AgentLoaderAttach.java:424)
        at com.instana.agent.loader.AgentLoaderAttach.run(AgentLoaderAttach.java:150)
        at com.instana.agent.loader.AgentLoaderAttach.parseArgsAndRun(AgentLoaderAttach.java:100)
        at com.instana.agent.loader.AgentLoaderAttach.main(AgentLoaderAttach.java:83)

This issue is only meant to be a question if you have seen this problem before?
I will try to debug the problem and provide a PR when i have an idea what might be wrong.

I don't see any major difference to how this is done with the tools.jar from the Hotspot JDK directly
https://github.com/AdoptOpenJDK/openjdk-jdk8u/blob/2544d2a351eca1a3d62276f969dd2d95e4a4d2b6/jdk/src/solaris/native/sun/tools/attach/SolarisVirtualMachine.c#L247

[raphw]

No, I have not seen that, unfortunately. To be honest, I installed and used Solaris a single time, just to make this work and have not used it since.

If you come up with a patch, I am happy to merge it!