Multple JDI tests failing on JDK8-s390x_Linux (scaffold issue?)

[M-Davies]

Describe the bug
In the nightlies, we have 9 tests that are consistently failing on JDK8 zlinux. I'll use this issue as a convenient dumping ground for them all as they all seem to be related

> com/sun/jdi/AllLineLocations.java.AllLineLocations
> com/sun/jdi/EarlyReturnTest.java.EarlyReturnTest
> com/sun/jdi/MethodExitReturnValuesTest.java.MethodExitReturnValuesTest
> com/sun/jdi/NativeInstanceFilter.java.NativeInstanceFilter
> com/sun/jdi/PopAndStepTest.java.PopAndStepTest
> com/sun/jdi/PopAsynchronousTest.java.PopAsynchronousTest
> com/sun/jdi/PopSynchronousTest.java.PopSynchronousTest
> com/sun/jdi/RedefineCrossStart.java.RedefineCrossStart
> com/sun/jdi/redefineMethod/RedefineTest.java.RedefineTest

To Reproduce
Steps to reproduce the behavior (or Grinder rerun link):

1. Go to https://ci.adoptopenjdk.net/job/Grinder
2. Execute a grinder with these params https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=8&JDK_IMPL=hotspot&BUILD_LIST=openjdk&PLATFORM=s390x_linux&TARGET=jdk_jdi_0
3. See error

Expected behavior
Tests pass

Screenshots
I've bundled these tests together as I they all utilise various scaffolding programs (JDI, Test and Shell) to set up the environment for the test.

Each of these are heavily dependant on the infrastructure of the machine that the test is running on and, as identified in #1711, there are some differences between versions and platforms that could cause the errors seen here. The purpose of this issue is to examine the scaffolding programs to see if there is something that is causing these failures and also to use as a placeholder for excluding these tests until they can be triaged appropriately.

Additional context

• Identified in our nightly build report Nightly Build&Test Triage Report temurin-build#1634 (comment)
• com/sun/jdi/PopAndStepTest.java and com/sun/jdi/PopSynchronousTest.java and com/sun/jdi/Redefine-g.sh all have a very similar error. Its likely fixing the scaffold behind each of these tests (Test) will resolve all three failures.

[M-Davies]

Exclude PR #1738

[karianna]

@M-Davies FYI you can actually link PR's on the RH side of a GitHub issue 'Linked Pull Requests'

[M-Davies]

@karianna Can this be reopened? The issue still persists, the PR was just to exclude it from the nightly testing so we don't keep seeing known failures :)

[M-Davies]

Discounting any infra problems, it seems likely it's something in this timeframe https://github.com/AdoptOpenJDK/openjdk-jdk8u/compare/master@%7B02-23-2020%7D...master@%7B04-09-2020%7D that is the cause of the failures since runs prior to 23rd Feb are green.

[M-Davies]

The range of commits above looks fine to me. I'll take a closer look at some of these scaffoding tests

[M-Davies]

Looking at TestScaffold specifically, there doesn't seem to be anything glaringly obvious that is wrong. This is used in EarlyReturnTest.java.

09:58:07  STDERR:
09:58:07  run args: [EarlyReturnTarg]
09:58:07  failure: byte: expected 42, got 0
09:58:07  failure: char: expected a, got 
09:58:07  failure: double: expected 6.6, got 2.2
09:58:07  failure: float: expected 9.9, got 3.3
09:58:07  failure: int: expected 7, got 1
09:58:07  failure: long: expected -9223372036854775808, got 9223372036854775807
09:58:07  failure: short: expected 3, got 0
09:58:07  failure: boolean: expected true, got false
09:58:07  failure: String: expected wxyz, got abc
09:58:07  failure: Class: expected instance of java.lang.Class(reflected class=java.lang.String, id=92), got instance of java.lang.Class(reflected class=java.lang.Object, id=88)
09:58:07  failure: ClassLoader: expected null, got instance of java.net.URLClassLoader(id=176)
09:58:07  failure: Thread: expected null, got instance of java.lang.Thread(name='main', id=1)
09:58:07  failure: ThreadgGroup: expected null, got instance of java.lang.ThreadGroup(name='main', id=178)
09:58:07  failure: in[2]: expected 12, got 3
09:58:07  failure: Object: expected EarlyReturnTarg.eobjectValue, got instance of java.util.Date(id=183)
09:58:07  failure: byte: expected 42, got 0
09:58:07  failure: char: expected a, got 
09:58:07  failure: double: expected 6.6, got 2.2
09:58:07  failure: float: expected 9.9, got 3.3
09:58:07  failure: int: expected 7, got 1
09:58:07  failure: long: expected -9223372036854775808, got 9223372036854775807
09:58:07  failure: short: expected 3, got 0
09:58:07  failure: boolean: expected true, got false
09:58:07  failure: String: expected wxyz, got abc
09:58:07  failure: in[2]: expected 12, got 3
09:58:07  failure: Class: expected instance of java.lang.Class(reflected class=java.lang.String, id=92), got instance of java.lang.Class(reflected class=java.lang.Object, id=88)
09:58:07  failure: ClassLoader: expected null, got instance of java.net.URLClassLoader(id=176)
09:58:07  failure: Thread: expected null, got instance of java.lang.Thread(name='main', id=1)
09:58:07  failure: ThreadgGroup: expected null, got instance of java.lang.ThreadGroup(name='main', id=178)
09:58:07  failure: Object: expected EarlyReturnTarg.eobjectValue, got instance of java.util.Date(id=183)
09:58:07  Note: Unexpected Debuggee Exception: java.lang.Exception at line 277
09:58:07  detailMessage = "EarlyReturnTarg: failed"
09:58:07  failure: Expected 34, but got 4
09:58:07  java.lang.Exception: EarlyReturnTest: failed
09:58:07  	at EarlyReturnTest.runTests(EarlyReturnTest.java:805)
09:58:07  	at TestScaffold.startTests(TestScaffold.java:429)
09:58:07  	at EarlyReturnTest.main(EarlyReturnTest.java:312)
09:58:07  	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
09:58:07  	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
09:58:07  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
09:58:07  	at java.lang.reflect.Method.invoke(Method.java:498)
09:58:07  	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
09:58:07  	at java.lang.Thread.run(Thread.java:748)
09:58:07  
09:58:07  JavaTest Message: Test threw exception: java.lang.Exception
09:58:07  JavaTest Message: shutting down test
09:58:07  
09:58:07  STDOUT:
09:58:07  JVM version:1.8.0_252
09:58:07  JDI version: 1.8
09:58:07  JVM description: Java Debug Interface (Reference Implementation) version 1.8 
09:58:07  Java Debug Wire Protocol (Reference Implementation) version 1.8
09:58:07  JVM Debug Interface version 1.2
09:58:07  JVM version 1.8.0_252 (OpenJDK 64-Bit Zero VM, interpreted mode, sharing)
09:58:07  debugee in main
09:58:07  debugee in doit with a debugger watching.  Early returns expected.
09:58:07  "==========  Testing static methods ================"
09:58:07  Cannot force early return for method: s_bytef
09:58:07  Cannot force early return for method: s_charf
09:58:07  Cannot force early return for method: s_doublef
09:58:07  Cannot force early return for method: s_floatf
09:58:07  Cannot force early return for method: s_intf
09:58:07  Cannot force early return for method: s_longf
09:58:07  Cannot force early return for method: s_shortf
09:58:07  Cannot force early return for method: s_booleanf
09:58:07  Cannot force early return for method: s_stringf
09:58:07  Cannot force early return for method: s_classf
09:58:07  Cannot force early return for method: s_classLoaderf
09:58:07  Cannot force early return for method: s_threadf
09:58:07  Cannot force early return for method: s_threadGroupf
09:58:07  Cannot force early return for method: s_intArrayf
09:58:07  Cannot force early return for method: s_nullObjectf
09:58:07  Passed: NullObject: null
09:58:07  Cannot force early return for method: s_objectf
09:58:07  Cannot force early return for method: s_voidf
09:58:07  debugee in s_voidf
09:58:07  Passed: Void
09:58:07  "==========  Testing instance methods ================"
09:58:07  Cannot force early return for method: i_bytef
09:58:07  Cannot force early return for method: i_charf
09:58:07  Cannot force early return for method: i_doublef
09:58:07  Cannot force early return for method: i_floatf
09:58:07  Cannot force early return for method: i_intf
09:58:07  Cannot force early return for method: i_longf
09:58:07  Cannot force early return for method: i_shortf
09:58:07  Cannot force early return for method: i_booleanf
09:58:07  Cannot force early return for method: i_stringf
09:58:07  Cannot force early return for method: i_intArrayf
09:58:07  Cannot force early return for method: i_classf
09:58:07  Cannot force early return for method: i_classLoaderf
09:58:07  Cannot force early return for method: i_threadf
09:58:07  Cannot force early return for method: i_threadGroupf
09:58:07  Cannot force early return for method: i_nullObjectf
09:58:07  Passed: NullObject: null
09:58:07  Cannot force early return for method: i_objectf
09:58:07  Cannot force early return for method: i_voidf
09:58:07  Passed: Void
09:58:07  Exception in thread "main" java.lang.Exception: EarlyReturnTarg: failed
09:58:07  	at EarlyReturnTarg.main(EarlyReturnTest.java:277)
09:58:07  All done, 4 passed
09:58:07  
09:58:07  EarlyReturnTest: failed

It is this line specifically that is throwing the errors as shown above. It appears the VM is not setup correctly, I'll see if I can spit out some debugging output to check the versions and capability of the VM.

[M-Davies]

16:45:51  [DEBUG] vmm.majorVersion is : 
16:45:51  1
16:45:51  [DEBUG] vmm.minorVersion is : 
16:45:51  8
16:45:51  [DEBUG] Can vm force an early return? : 
16:45:51  false

So canForceEarlyReturn() is not what it should be.

• canForceEarlyReturn(), vm (virtual machine) is declared here: https://github.com/AdoptOpenJDK/openjdk-jdk8u/blob/981f13d4305f45a6d4ad47428fb6994edd22dbb2/hotspot/agent/test/jdi/TestScaffold.java#L505
• More documentation on Oracle's side can be seen here https://docs.oracle.com/javase/8/docs/jdk/api/jpda/jdi/com/sun/jdi/ThreadReference.html#forceEarlyReturn-com.sun.jdi.Value-

[M-Davies]

That does bring into question the output further up in the test when it pumps out the version of the JVM version. I wonder where it is getting this JVM information from.

16:45:51  JVM version:1.8.0_252
16:45:51  JDI version: 1.8
16:45:51  JVM description: Java Debug Interface (Reference Implementation) version 1.8 
16:45:51  Java Debug Wire Protocol (Reference Implementation) version 1.8
16:45:51  JVM Debug Interface version 1.2
16:45:51  JVM version 1.8.0_252 (OpenJDK 64-Bit Zero VM, interpreted mode, sharing)

It would be good to compare this test against a machine that isn't a test-marist-ubuntu1604-s390x-1/2/3/4 one to check if it's a environment issue.

Running a grinder on our s390x machine that isn't test-marist doesn't seem to be configured correctly for grinders https://ci.adoptopenjdk.net/job/Grinder/2916/console.

I'll see if I can get hold of one on Fyre

[M-Davies]

Ran a couple of internal grinders since fyre appears to be down. Both failed in the same way.

@smlambert could you take a look at this?

[M-Davies]

Ran a modified version of the failing line without the canForceEarlyReturn() to see what exception was thrown.

java.lang.UnsupportedOperationException: target does not support the forcing of a method to return early

Cross checking with https://docs.oracle.com/javase/8/docs/jdk/api/jpda/jdi/com/sun/jdi/ThreadReference.html#forceEarlyReturn-com.sun.jdi.Value-, this rules out a few options.

I have also noticed that there are two references to vm() in the code. I'll try and find out which one is being used

• https://github.com/AdoptOpenJDK/openjdk-jdk8u/blob/6f07d2af61974a9550ad55c28309ab8e6bb65790/hotspot/agent/test/jdi/TestScaffold.java#L505 ❎
• https://github.com/AdoptOpenJDK/openjdk-jdk8u/blob/6f07d2af61974a9550ad55c28309ab8e6bb65790/jdk/test/com/sun/jdi/TestScaffold.java#L639 ✅ Its this one

[M-Davies]

These have not been seen recently and my grinder passed https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/7653/console so assuming these have all been fixed

[sophia-guo]

Reopen as test has been excluded. The grinder without test re-enabled doesn't means the issue has been fixed.