JVM_EnqueueOperation() stubbed #3441

planetf1 · 2018-10-25T07:15:02Z

When trying to build github.com/odpi/egeria with 'mvn clean install' using maven 3.5.4 on windows 10, within IntelliJ IDEA, I get:

Microsoft Visual C++ Runtime Library
X Assertion failed!

Program: c:\java\jdku192-b12\bin\java.exe
File: j7vmi.c
Line: 2713

Expression: !"JVM_EnqueueOperation() stubbed!"

The JDK was downloaded from adoptopenjdk.net

I can ignore the popup, and the process does continue

A similar issue occurs on MacOS, but the exception is different:

planetf1 · 2018-10-25T07:21:50Z

See also #3442 for a similar dump on macOS - but with a different exception

DanHeidinga · 2018-10-31T14:31:38Z

@ChengJin01 Can you look at this?

ChengJin01 · 2018-10-31T14:32:58Z

Will investigate to see what happened in there.

ChengJin01 · 2018-10-31T17:14:46Z

I don't think the issue here has anything to do with #3442 as JVM_SocketShutdown() was already implemented on UNIX-based system (So it just got enabled for OSX).

JVM_EnqueueOperation() was not implemented on any platform. So I need to figure out how this got invoked on Windows first.

pshipton · 2018-10-31T19:59:12Z

It seems something is trying to use the OpenJDK/Hotspot attach mechanism instead of the OpenJ9 one. i.e. it must be hard coded to use sun.tools.attach.WindowsVirtualMachine (or similar concrete class) instead of using the generic api classes as it should.

ChengJin01 · 2018-10-31T20:18:43Z

I already noticed that at openj9-openjdk-jdk8/jdk/src/windows/native/sun/tools/attach/WindowsVirtualMachine.c.

/* exported function in target VM */
typedef jint (WINAPI* EnqueueOperationFunc)
    (const char* cmd, const char* arg1, const char* arg2, const char* arg3, const char* pipename);

/*
 * Class:     sun_tools_attach_WindowsVirtualMachine
 * Method:    enqueue
 * Signature: (JZLjava/lang/String;[Ljava/lang/Object;)V
 */
JNIEXPORT void JNICALL Java_sun_tools_attach_WindowsVirtualMachine_enqueue
  (JNIEnv *env, jclass cls, jlong handle, jbyteArray stub, jstring cmd,
   jstring pipename, jobjectArray args)
{...

If so, it looks like there are some setting to be modified so as to avoid sun.tools.attach.WindowsVirtualMachine in favor of our build.

ChengJin01 · 2018-10-31T20:28:10Z

@planetf1, I already installed maven 3.5.4 and IntelliJ IDEA on Windows 10 but failed to compile after building with "mvn clean install" on the imported egeria project, which I suspect some setting went wrong before compilation.
egeria_install_logs.txt

Could you provide the specific steps on how to reproduce the assertion?

pshipton · 2018-10-31T20:34:22Z

it looks like there are some setting to be modified so as to avoid sun.tools.attach.WindowsVirtualMachine in favor of our build.

Note the OpenJ9 build is configured correctly, I think this problem would only occur if something is hard coded to refer to the Hotspot attach api implementation rather than the generic api classes https://docs.oracle.com/javase/8/docs/jdk/api/attach/spec/index.html

ChengJin01 · 2018-10-31T20:39:45Z

I think this problem would only occur if something is hard coded to refer to the Hotspot attach API implementatation...

That should come from the application level. But I didn't find anything related to sun_tools_attach_WindowsVirtualMachine in https://github.com/odpi/egeria.

Probably it was triggered from some loading plugins/packages.

pshipton · 2018-10-31T21:06:35Z

For reference, here is the configuration that selects the OpenJ9 attach api implementation for the generic api.

jdk8:
https://github.com/ibmruntimes/openj9-openjdk-jdk8/blob/openj9/jdk/src/share/classes/sun/tools/attach/META-INF/services/com.sun.tools.attach.spi.AttachProvider

jdk11:
https://github.com/ibmruntimes/openj9-openjdk-jdk11/blob/openj9/src/jdk.attach/share/classes/module-info.java - modified to remove sun.tools.attach.AttachProviderImpl so the following can take affect
https://github.com/eclipse/openj9/blob/master/jcl/src/jdk.attach/share/classes/module-info.java.extra

pshipton · 2018-10-31T21:07:23Z

try looking for sun.tools.attach.AttachProviderImpl (jdk11) instead

pshipton · 2018-10-31T21:08:15Z

or sun.tools.attach.WindowsAttachProvider (jdk8)

ChengJin01 · 2018-10-31T21:09:13Z

I just found WindowsVirtualMachine comes from lib/tools.jar in jdk8u192-b12

/cygdrive/c/temp_jdk8/jdk8u192-b12/lib/tools/sun/tools/attach (extracted from lib/tools.jar)
$ ls -l
total 32
-rwxrwx---+ 1 Administrators None 548 Oct 19 15:37 'HotSpotAttachProvider$HotSpotVirtualMachineDescriptor.class'
-rwxrwx---+ 1 Administrators None 3538 Oct 19 15:37 HotSpotAttachProvider.class
-rwxrwx---+ 1 Administrators None 8440 Oct 19 15:37 HotSpotVirtualMachine.class
-rwxrwx---+ 1 Administrators None 3544 Oct 19 15:37 WindowsAttachProvider.class
-rwxrwx---+ 1 Administrators None 1060 Oct 19 15:37 'WindowsVirtualMachine$PipedInputStream.class'
-rwxrwx---+ 1 Administrators None 3217 Oct 19 15:37 WindowsVirtualMachine.class

pshipton · 2018-10-31T22:43:17Z

Right, the files are there, but not used by the api classes because of the settings in #3441 (comment) . These classes would only be used if referenced directly by some code.

ChengJin01 · 2018-10-31T23:36:29Z

The only way to prove that is to reproduce locally to see what happens on the backtraces; otherwise, I suspect the setting at openj9-openjdk-jdk8 might go wrong in forwarding to the correct OpenJ9 class (/jcl/src/jdk.attach/share/classes/com/ibm/tools/attach/attacher/OpenJ9VirtualMachine)

ChengJin01 · 2018-11-01T22:07:55Z

After adjusting the setting with maven, I managed to trigger the same assertion failures as follows:

15  Id: 9abc.19e9c Suspend: 0 Teb: 00000000`006c9000 Unfrozen
 # Child-SP          RetAddr           Call Site
00 00000000`137ae3b8 00007ffc`9d5a3b7f ntdll!NtWaitForSingleObject+0x14
01 00000000`137ae3c0 00007ffc`80ca0fcd KERNELBASE!WaitForSingleObjectEx+0x9f
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for j9dmp29.dll - 
02 00000000`137ae460 00007ffc`81ca4cf4 j9prt29!j9port_init_library+0x1e5fd
03 00000000`137ae500 00007ffc`81ca49c5 j9dmp29+0x4cf4
04 00000000`137ae560 00007ffc`80c9d9b6 j9dmp29+0x49c5
05 00000000`137ae590 00007ffc`80c9dbb4 j9prt29!j9port_init_library+0x1afe6
06 00000000`137ae5d0 00007ffc`81ca5fbf j9prt29!j9port_init_library+0x1b1e4
07 00000000`137ae7b0 00007ffc`81ca6280 j9dmp29+0x5fbf
08 00000000`137ae820 00007ffc`81cb83c2 j9dmp29+0x6280
09 00000000`137aed00 00007ffc`81ca7884 j9dmp29!J9VMDllMain+0xf182
0a 00000000`137af070 00000000`624f0ef1 j9dmp29+0x7884
0b 00000000`137af0c0 00000000`624f7255 msvcr100!raise+0x1ed [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\winsig.c @ 593] 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for jvm.dll - 
0c 00000000`137af120 00007ffc`827a793e msvcr100!_wassert+0x649 [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\assert.c @ 319] 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for jvm.dll - 
0d 00000000`137af800 00007ffc`8286468b jvm_7ffc827a0000!JVM_EnqueueOperation+0x1e
0e 00000000`137af830 00000000`0293007b jvm!JVM_EnqueueOperation+0x1b <--------------
0f 00000000`137af870 00000000`004e0000 0x293007b
10 00000000`137af878 00007ffc`8286466f 0x4e0000
11 00000000`137af880 00000000`00000000 jvm!JVM_DefineClassWithSourceCond+0x5f <------------

javacore.20181101.142234.25064.0002.txt

Looking at the code at /runtime/j9vm/j7vmi.c

jobject JNICALL
JVM_DefineClassWithSourceCond(jint arg0, jint arg1, jint arg2, jint arg3, jint arg4, jint arg5, jint arg6, jint arg7)
{
	assert(!"JVM_DefineClassWithSourceCond() stubbed!");
	return NULL;
}

It looks like it has nothing to do with AttachProvider/VirtualMachine at this point. (any connection with ClassLoader ?)

ChengJin01 · 2018-11-02T00:11:04Z

Looking at the command line option:
2CIUSERARG -javaagent:C:\JetBrains\IntelliJ IDEA 183.4139.22\lib\idea_rt.jar=60529:C:\JetBrains\IntelliJ IDEA 183.4139.22\bin

I suspect JVM_DefineClassWithSourceCond was invoked via Instrumentation from IntelliJ IDEA. So I double-check the directory at IntelliJ IDEA:

/cygdrive/c/JetBrains/IntelliJ IDEA 183.4139.22
$ grep -r JVM_DefineClassWithSourceCond
Binary file jre64/bin/server/jvm.dll matches

/cygdrive/c/JetBrains/IntelliJ IDEA 183.4139.22
$ grep -r JVM_EnqueueOperation
Binary file jre64/bin/attach.dll matches
Binary file jre64/bin/server/jvm.dll matches

Dump file jvm.dll (via dumpbin.exe)

 Section contains the following exports for jvm.dll

    ordinal hint RVA      name
       2699  A8A 0016CDE0 JVM_DefineClassWithSourceCond
    ...
       2706  A91 00265ED0 JVM_EnqueueOperation

With the backtrace above, it seems the calling path is : JVM_DefineClassWithSourceCond(IntelliJ) -> JVM_EnqueueOperation(IntelliJ) --> JVM_EnqueueOperation(OpenJ9)

pshipton · 2018-11-02T01:08:57Z

I suspect the native stack is wrong in showing JVM_DefineClassWithSourceCond. There is a second process involved which is causing the JVM_EnqueueOperation in this VM. This other process is using sun.tools.attach.WindowsVirtualMachine instead of the OpenJ9 attach api.

pshipton · 2018-11-02T01:10:41Z

If the only JVMs involved are OpenJ9, you can use the IBM_JAVA_OPTIONS environment variable to set -Xdump options to create a javacore when the sun.tools.attach.WindowsVirtualMachine code is called.

pshipton · 2018-11-02T01:17:43Z

Make that an -Xdump option when the sun.tools.attach.WindowsVirtualMachine class is loaded, which is probably good enough, or an -Xtrace option when specific code is called.

pshipton · 2018-11-02T01:18:39Z

fyi https://www.eclipse.org/openj9/docs/tool_builder/

pshipton · 2018-11-02T01:19:33Z

or more specifically
https://www.eclipse.org/openj9/tools/xdump_option_builder.html
https://www.eclipse.org/openj9/tools/xtrace_option_builder.html

ChengJin01 · 2018-11-02T03:06:24Z

I suspect the native stack is wrong in showing JVM_DefineClassWithSourceCond. There is a second process..

From the console in IntelliJ IDEA, it seems there is only one JVM getting started to build egeria

C:\temp_jdk8\jdk8u192-b12\bin\java.exe -Dmaven.multiModuleProjectDirectory=C:\temp_jdk8\egeria -Dmaven.home=C:\temp_jdk8\apache-maven-3.5.4 -Dclassworlds.conf=C:\temp_jdk8\apache-maven-3.5.4\bin\m2.conf "-javaagent:C:\JetBrains\IntelliJ IDEA 183.4139.22\lib\idea_rt.jar=62168:C:\JetBrains\IntelliJ IDEA 183.4139.22\bin" -Dfile.encoding=UTF-8 -classpath C:\temp_jdk8\apache-maven-3.5.4\boot\plexus-classworlds-2.5.2.jar org.codehaus.classworlds.Launcher -Didea.version=2018.3 clean install
JVMDUMP039I Processing dump event "abort", detail "" at 2018/11/01 20:00:59 - please wait.
JVMDUMP032I JVM requested System dump using 'C:\temp_jdk8\egeria\core.20181101.200059.60888.0001.dmp' in response to an event
...

I am not too sure whether a second process exists in the background. Probably need to compile a build to halt in JVM_EnqueueOperation() to see what happens.

pshipton · 2018-11-02T03:21:37Z

Well if you set IBM_JAVA_OPTIONS=-Xdump:java:events=load,filter=sun/tools/attach/WindowsVirtualMachine,request=exclusive+preempt you can see if a javacore gets created.

pshipton · 2018-11-02T03:24:10Z

Also, if you delete jre64/bin/attach.dll does the problem still occur?

ChengJin01 · 2018-11-02T03:46:50Z

if you set IBM_JAVA_OPTIONS=-Xdump:java:events=load,...

It seems IBM_JAVA_OPTIONS didn't get added to the command line (need to double-check the setting in maven and IntelliJ IDEA)

ChengJin01 · 2018-11-02T05:23:48Z

if you delete jre64/bin/attach.dll does the problem still occur?

I double-checked the steps:

tried to remove jre64/bin/attach.dll but failed (still in use in IntelliJ IDEA)
exit IntelliJ IDEA
removed jre64/bin/attach.dll and launched IntelliJ IDEA
ran "clean install" via "Execute Maven Goal"
the assertion failure disappeared.
exit IntelliJ IDEA and added attach.dll back to jre64/bin
launched IntelliJ IDEA and ran "clean install" via "Execute Maven Goal"
the assertion failure re-occurred.

Given that jre64/bin/attach.dll in IntelliJ IDEA is related to sun.tools.attach.WindowsVirtualMachine, I suspect a second process did exist but exit right after the assert failure occurred (just noticed that on the Task Manager)

ChengJin01 · 2018-11-02T21:43:26Z

if you set IBM_JAVA_OPTIONS=-Xdump:java:events=load,...

Already added it to the maven setting but no java core was generated.
I also compiled a build with printing messages in Java_sun_tools_attach_WindowsVirtualMachine_enqueue (which invokes JVM_EnqueueOperation) but nothing happened during execution.

Other interesting findings:

the problem only occurs by pressing the "Execute Maven Goal" button to build the maven project with the following command:

...bin\java.exe -Dmaven.multiModuleProjectDirectory=C:\temp_jdk8\egeria -Xdump:java:events=load,filter=sun/tools/attach/WindowsVirtualMachine,request=exclusive+preempt -Dmaven.home=C:\temp_jdk8\apache-maven-3.5.4 -Dclassworlds.conf=C:\temp_jdk8\apache-maven-3.5.4\bin\m2.conf "-javaagent:C:\JetBrains\IntelliJ IDEA 183.4139.22\lib\idea_rt.jar=65068:C:\JetBrains\IntelliJ IDEA 183.4139.22\bin" -Dfile.encoding=UTF-8 -classpath C:\temp_jdk8\apache-maven-3.5.4\boot\plexus-classworlds-2.5.2.jar org.codehaus.classworlds.Launcher -Didea.version=2018.3 clean install

the problem never happens if running the command above via IntelliJ IDEA Terminal console or directly via Windows command Terminal.

So I suspect the problem has nothing to do with this project but with any maven-based project which has to be built via the "Execute Maven Goal" button. I am not sure how this button works in IntelliJ IDEA but I suspect it first launched a JVM process (assert failure occurred in this process) which creates another JVM process to build the maven project. That explains why the maven project keeps being built even though the assert failure occurs.

pshipton · 2018-11-02T23:15:03Z

You could try removing the OpenJDK/Hotspot attach api classes like sun.tools.attach.WindowsVirtualMachine and see if this changes the behavior. Also check if there is a log file where errors/exceptions get logged.

pshipton · 2018-11-02T23:16:21Z

and you can go ahead and create a PR to do #3441 (comment)

ChengJin01 · 2018-11-08T18:11:28Z

I already created a PR at ibmruntimes/openj9-openjdk-jdk8#202 to exclude WindowsVirtualMachine related classes (attach.dll never generated during compilation). The assertion failure still occurred with the compiled build in IntelliJ.

C:\temp_jdk8\build_mingw_64_remove_attach_v1\windows-x86_64-normal-server-release\images\j2sdk-image\bin\java.exe -Dmaven.multiModuleProjectDirectory=C:\temp_jdk8\egeria -Xdump:java:events=load,filter=sun/tools/attach/WindowsVirtualMachine,request=exclusive+preempt -Dmaven.home=C:\temp_jdk8\apache-maven-3.5.4 -Dclassworlds.conf=C:\temp_jdk8\apache-maven-3.5.4\bin\m2.conf "-javaagent:C:\JetBrains\IntelliJ IDEA 183.4139.22\lib\idea_rt.jar=50872:C:\JetBrains\IntelliJ IDEA 183.4139.22\bin" -Dfile.encoding=UTF-8 -classpath C:\temp_jdk8\apache-maven-3.5.4\boot\plexus-classworlds-2.5.2.jar org.codehaus.classworlds.Launcher -Didea.version=2018.3 clean install
JVMDUMP039I Processing dump event "abort", detail "" at 2018/11/08 09:20:23 - please wait.
...

The result confirms that JVM_EnqueueOperation was not invoked from JDK8/OpenJ9.
Given that the assertion failure disappeared after removing jre64/bin/attach.dll from IntelliJ, I suspect that attach.dll from IntelliJ was still loaded to VM.

ChengJin01 · 2018-11-12T21:14:18Z

@pshipton, I looked over the issue you previously detected/closed at #2275 and I believe this is the exactly same problem as what we had here. It already explained what happened to the native stack:

...bundles sun.tools.attach.WindowsVirtualMachine.class which is specific to Hotspot. 
The enqueue() native is calling JVM_EnqueueOperation()...

which is the same result we encountered in IntelliJ. So the backtrace collected previously was correct that the invocation of JVM_EnqueueOperation came from IntelliJ.

To verify the workaround suggested at #2275, I replaced jre64/lib/tools.jar in IntelliJ with tools.jar in our jdk8/openj9 build and it works good without the assertion failure box. So there is nothing more we can do on our side at this point.

We can still remove the attach api classes with WindowsVirtualMachine via ibmruntimes/openj9-openjdk-jdk8#202 if you agree even though it has nothing to do with this issue.

pshipton · 2018-11-12T21:46:17Z

@ChengJin01 I think we should still proceed with ibmruntimes/openj9-openjdk-jdk8#202

ChengJin01 · 2018-11-12T21:52:53Z

Will verify the changes with WindowsVirtualMachine at ibmruntimes/openj9-openjdk-jdk8#202 after updating.

In addition, this issue here should be marked as duplicate with #2275

pshipton · 2018-11-12T21:56:13Z

Similarly to #2275, we should contact IntelliJ and ask they include the OpenJ9 tools.jar in addition to the Hotspot version, and use the appropriate tools.jar that matches the JVM.

ChengJin01 · 2018-11-12T22:36:52Z

Request has been created at https://intellij-support.jetbrains.com/hc/en-us/requests/1864909 to track the issue.

ChengJin01 · 2018-11-13T15:37:45Z

The request was reported to https://youtrack.jetbrains.com/issue/JRE-1037

kgrechishchev · 2018-11-15T11:40:48Z

@ChengJin01 @pshipton

I think I have found another way to reproduce the same issue.
Note, that I do not use IntellJ Idea here, it is just mvn project

Please check simple 1 class maven project
attachapitest.zip
You can reproduce the issue by running it using mvn test

It fails with exactly the same error message.

I use power org.powermock.modules.junit4.rule.PowerMockRule; class in the test cases. It seems it uses attach.dll. Actually org.powermock.reflect.internal.WhiteboxImpl uses it.

If I just remove attach.dll, I face (expected actually) UnsatisfiedLinkError:

java.lang.ExceptionInInitializerError
        at java.lang.J9VMInternals.ensureError(J9VMInternals.java:146)
        at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:135)
        at PowerMockRuleTest.<init>(PowerMockRuleTest.java:9)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:217)
        at org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:266)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:263)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
        at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
        at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

Caused by: java.lang.IllegalStateException: Native library for Attach API not available in this JRE
        at org.powermock.modules.agent.AgentLoader.getVirtualMachineImplementationFromEmbeddedOnes(AgentLoader.java:126)
        at org.powermock.modules.agent.AgentLoader.loadAgent(AgentLoader.java:81)
        at org.powermock.modules.agent.AgentInitialization.initializeAccordingToJDKVersion(AgentInitialization.java:40)
        at org.powermock.modules.agent.PowerMockAgent.initializeIfNeeded(PowerMockAgent.java:91)
        at org.powermock.modules.agent.PowerMockAgent.initializeIfPossible(PowerMockAgent.java:105)
        at org.powermock.modules.junit4.rule.PowerMockRule.<clinit>(PowerMockRule.java:37)
        ... 29 more
Caused by: java.lang.UnsatisfiedLinkError: attach (Not found in java.library.path)
        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1434)
        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1404)
        at java.lang.System.loadLibrary(System.java:540)
        at sun.tools.attach.WindowsVirtualMachine.<clinit>(WindowsVirtualMachine.java:185)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.powermock.reflect.internal.WhiteboxImpl.createInstance(WhiteboxImpl.java:1428)
        at org.powermock.reflect.internal.WhiteboxImpl.invokeConstructor(WhiteboxImpl.java:1301)
        at org.powermock.reflect.Whitebox.invokeConstructor(Whitebox.java:497)
        at org.powermock.modules.agent.AgentLoader.getVirtualMachineImplementationFromEmbeddedOnes(AgentLoader.java:120)
        ... 34 more

My env.
OS: Windows 7 x64
Java and maven:

d:\src\attachapitest>mvn -version
Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T21:41:47+03:00)
Maven home: C:\maven\bin\..
Java version: 1.8.0_192, vendor: Eclipse OpenJ9, runtime: c:\Java\jdk1.8-32-J9\jre
Default locale: en_US, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "x86", family: "windows"

d:\src\attachapitest>java -version
openjdk version "1.8.0_192"
OpenJDK Runtime Environment (build 1.8.0_192-b12)
Eclipse OpenJ9 VM (build openj9-0.11.0, JRE 1.8.0 Windows 7 x86-32-Bit 20181019_40 (JIT enabled, AOT enabled)
OpenJ9   - 090ff9dc
OMR      - ea548a66
JCL      - 51609250b5 based on jdk8u192-b12)

Does it mean that power mockito depends on hotspot version and will not work with J9?

ChengJin01 · 2018-11-15T15:40:30Z

Apparently it was still using sun.tools.attach.WindowsVirtualMachine (sun.tools.attach.WindowsVirtualMachine.class is specific to Hotspot. The enqueue() native is calling JVM_EnqueueOperation().) instead of com.ibm.tools.attach.attacher.OpenJ9VirtualMachine. That's why it triggered the same assertion failure.

...
at sun.tools.attach.WindowsVirtualMachine.<clinit>(WindowsVirtualMachine.java:185) <-------
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.powermock.reflect.internal.WhiteboxImpl.createInstance(WhiteboxImpl.java:1428)
        at org.powermock.reflect.internal.WhiteboxImpl.invokeConstructor(WhiteboxImpl.java:1301)
        at org.powermock.reflect.Whitebox.invokeConstructor(Whitebox.java:497)
        at org.powermock.modules.agent.AgentLoader.getVirtualMachineImplementationFromEmbeddedOnes(AgentLoader.java:120)

It looks like this is another issue. Need to figure out why it happens in this way.

kgrechishchev · 2018-11-15T16:53:05Z

Let me know if you want me to open separate issue

ChengJin01 · 2018-11-15T18:32:36Z

@kgrechishchev , please show me the steps (what else needs to be installed) to reproduce the issue because didn't see any assertion failure when running "mvn test" except building failure.

C:\temp_jdk8\attachapitest>mvn -version
Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T11:41:47-07:00)
Maven home: C:\temp_jdk8\apache-maven-3.6.0\bin\..
Java version: 1.8.0_192-internal, vendor: Eclipse OpenJ9, runtime: C:\temp_jdk8\jdk8_mingw_v1_v1\windows-x86_64-normal-server-release\images\j2sdk-image\jre
Default locale: en_US, platform encoding: Cp1252
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

C:\temp_jdk8\attachapitest>mvn test
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< org.eclipse.openj9:attach-api-test >-----------------
[INFO] Building attach-api-test 1.0
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from central: https://repo.maven.apache.org/maven2/org/powermock/powermock-module-junit4-rule-agent/1.6.4/powermock-module-junit4-rule-agent-1.6.4.pom
Downloading from central: https://repo.maven.apache.org/maven2/org/powermock/powermock-api-mockito/1.6.4/powermock-api-mockito-1.6.4.pom
Downloading from central: https://repo.maven.apache.org/maven2/org/powermock/powermock-module-junit4/1.6.4/powermock-module-junit4-1.6.4.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.876 s
[INFO] Finished at: 2018-11-15T10:28:46-08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project attach-api-test: Could not resolve dependencies for project org.eclipse.openj9:attach-api-test:jar:1.0: Failed to collect dependencies at org.powermock:powermock-module-junit4-rule-agent:jar:1.6.4: Failed to read artifact descriptor for org.powermock:powermock-module-junit4-rule-agent:jar:1.6.4: Could not transfer artifact org.powermock:powermock-module-junit4-rule-agent:pom:1.6.4 from/to central (https://repo.maven.apache.org/maven2): java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

ChengJin01 · 2018-11-15T19:51:41Z

I double-checked the source code at https://github.com/powermock/powermock as follows:
powermock/powermock-modules/powermock-module-javaagent/src/main/java/org/powermock/modules/agent/AgentLoader.java

    private VirtualMachine getVirtualMachineImplementationFromEmbeddedOnes() {
        try {
            Class<? extends VirtualMachine> vmClass;

            if (File.separatorChar == '\\') {
                vmClass = WindowsVirtualMachine.class; 
<---- their code bundles sun.tools.attach.WindowsVirtualMachine to be reflected.
            } else {
                String osName = System.getProperty("os.name");

                if (osName.startsWith("Linux") || osName.startsWith("LINUX")) {
                    vmClass = LinuxVirtualMachine.class;
                } else if (osName.startsWith("Mac OS X")) {
                    vmClass = BsdVirtualMachine.class;
                } else if (osName.startsWith("Solaris")) {
                    vmClass = SolarisVirtualMachine.class;
                } else {
                    return null;
                }
            }

            // This is only done with Reflection to avoid the JVM pre-loading all the XyzVirtualMachine classes.
            Class<?>[] parameterTypes = {AttachProvider.class, String.class};

            VirtualMachine newVM = null;
            try {
                newVM = Whitebox.invokeConstructor(vmClass, parameterTypes, new Object[]{ATTACH_PROVIDER, pid});
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
            return newVM;
        } catch (UnsatisfiedLinkError e) {
            throw new IllegalStateException("Native library for Attach API not available in this JRE", e);
        }
    }

It turns out the code there bundles WindowsVirtualMachine to trigger JVM_EnqueueOperation eventually by using reflection on WindowsVirtualMachine, which can't be prevented on our side as OpenJ9 still includes WindowsVirtualMachine without any use for now.

Thus, it will throw out an RuntimeException once WindowsVirtualMachine is removed via ibmruntimes/openj9-openjdk-jdk8#202.

To get their code work on OpenJ9, they need to update getVirtualMachineImplementationFromEmbeddedOnes() to support OpenJ9VirtualMachine in addition to WindowsVirtualMachine.

The change is to remove these api classes and stop them from generating attach.dll and bytecode on Windows as these classes are replaced with the implementation in OpenJ9. Related to eclipse-openj9/openj9#3441 Signed-off-by: CHENGJin <jincheng@ca.ibm.com>

The test change is to replace the attach library with another native library as attach library has been removed. Related to eclipse-openj9#3441 Signed-off-by: CHENGJin <jincheng@ca.ibm.com>

kgrechishchev · 2019-02-06T09:33:47Z

@ChengJin01 @DanHeidinga
Thanks for fixing the error message, it looks much better now and provides a better user experience.

I have tested it with the latest versions of powerkmock and J9.

I have logged an issue on powermock/powermock#970 You can see how it looks now, it basically prints java.lang.IllegalStateException: Native library for Attach API not available in this JRE

planetf1 · 2019-04-26T09:42:38Z

I can confirm that when using IntelliJ (latest EAP) with the latest J9 Java 8 JVM, I no longer see this issue and build/compile works normally. Thanks!

planetf1 mentioned this issue Oct 25, 2018

JVM_SocketShutdown() stubbed #3442

Closed

pshipton added the userRaised label Oct 25, 2018

DanHeidinga added the comp:vm label Oct 31, 2018

ChengJin01 mentioned this issue Nov 8, 2018

Remove OpenJDK/Hotspot attach api classes ibmruntimes/openj9-openjdk-jdk8#202

Merged

planetf1 mentioned this issue Nov 13, 2018

Move from Oracle JDK to (adopt) OpenJDK odpi/egeria#228

Closed

ChengJin01 mentioned this issue Nov 16, 2018

Remove the attach library in test #3720

Merged

This was referenced Nov 22, 2018

Microsoft Visual C++ Runtime Assertion Failed adoptium/temurin-build#760

Closed

Windows: Assertion Failed on latest OpenJ9 JDK11 code #3805

Closed

DanHeidinga mentioned this issue Jan 28, 2019

windows10&openj9-jdk8??JVM_EnqueueOperation() stubbed #4465

Closed

kgrechishchev mentioned this issue Feb 6, 2019

Support Java8 openJ9 JDK powermock/powermock#970

Open

sharon-wang mentioned this issue Feb 25, 2019

Update assertion error message for JVM_EnqueueOperation() #4871

Merged

pshipton closed this as completed Apr 12, 2019

lostsnow mentioned this issue Apr 24, 2023

Attach to OpenJ9 VM cause Assertion failed on Windows jattach/jattach#71

Closed

JVM_EnqueueOperation() stubbed #3441

JVM_EnqueueOperation() stubbed #3441

Comments

planetf1 commented Oct 25, 2018

planetf1 commented Oct 25, 2018 • edited

DanHeidinga commented Oct 31, 2018

ChengJin01 commented Oct 31, 2018

ChengJin01 commented Oct 31, 2018

pshipton commented Oct 31, 2018 • edited

ChengJin01 commented Oct 31, 2018 • edited

ChengJin01 commented Oct 31, 2018 • edited

pshipton commented Oct 31, 2018 • edited

ChengJin01 commented Oct 31, 2018 • edited

pshipton commented Oct 31, 2018

pshipton commented Oct 31, 2018 • edited

pshipton commented Oct 31, 2018 • edited

ChengJin01 commented Oct 31, 2018

pshipton commented Oct 31, 2018

ChengJin01 commented Oct 31, 2018

ChengJin01 commented Nov 1, 2018 • edited

ChengJin01 commented Nov 2, 2018 • edited

pshipton commented Nov 2, 2018

pshipton commented Nov 2, 2018

pshipton commented Nov 2, 2018

pshipton commented Nov 2, 2018

pshipton commented Nov 2, 2018

ChengJin01 commented Nov 2, 2018

pshipton commented Nov 2, 2018

pshipton commented Nov 2, 2018

ChengJin01 commented Nov 2, 2018 • edited

ChengJin01 commented Nov 2, 2018 • edited

ChengJin01 commented Nov 2, 2018

pshipton commented Nov 2, 2018

pshipton commented Nov 2, 2018

ChengJin01 commented Nov 8, 2018 • edited

ChengJin01 commented Nov 12, 2018 • edited

pshipton commented Nov 12, 2018

ChengJin01 commented Nov 12, 2018

pshipton commented Nov 12, 2018

ChengJin01 commented Nov 12, 2018

ChengJin01 commented Nov 13, 2018

kgrechishchev commented Nov 15, 2018 • edited

ChengJin01 commented Nov 15, 2018

kgrechishchev commented Nov 15, 2018

ChengJin01 commented Nov 15, 2018 • edited

ChengJin01 commented Nov 15, 2018 • edited

kgrechishchev commented Feb 6, 2019

planetf1 commented Apr 26, 2019

planetf1 commented Oct 25, 2018 •

edited

pshipton commented Oct 31, 2018 •

edited

ChengJin01 commented Oct 31, 2018 •

edited

ChengJin01 commented Oct 31, 2018 •

edited

pshipton commented Oct 31, 2018 •

edited

ChengJin01 commented Oct 31, 2018 •

edited

pshipton commented Oct 31, 2018 •

edited

pshipton commented Oct 31, 2018 •

edited

ChengJin01 commented Nov 1, 2018 •

edited

ChengJin01 commented Nov 2, 2018 •

edited

ChengJin01 commented Nov 2, 2018 •

edited

ChengJin01 commented Nov 2, 2018 •

edited

ChengJin01 commented Nov 8, 2018 •

edited

ChengJin01 commented Nov 12, 2018 •

edited

kgrechishchev commented Nov 15, 2018 •

edited

ChengJin01 commented Nov 15, 2018 •

edited

ChengJin01 commented Nov 15, 2018 •

edited