Enable downcall & upcall test suites for Java 17/18/19 #15310

ChengJin01 · 2022-06-13T18:32:41Z

The changes include all test suites intended for downcall
and upcall in Java17, 18 and 19 which can be enabled
on the following platforms:
1)downcall: all supported platforms
2)upcall: AIX/PPC and Linux/PPC64LE

Signed-off-by: Cheng Jin jincheng@ca.ibm.com

ChengJin01 · 2022-06-13T18:32:56Z

The change also belongs to the issue intended for the upcall work at #15068.

Note:

the test suites cover the downcall & upcall tests in Java 19 and upcall specific tests in Java 17 & 18.
the PR depends on all code included in Design and Implementation of Foreign Linker API: Upcall #15068. So the code there must be merged at first before the test suites.

Reviewer: @tajila, @llxia (for the test framework)
FYI: @DanHeidinga, @pshipton, @gacholio

tajila · 2022-06-14T13:43:53Z

@llxia Please review these changes

llxia

test changes look good to me

tajila · 2022-06-17T17:32:31Z

jenkins test sanity amac jdk17

tajila · 2022-06-17T17:32:44Z

jenkins test sanity plinux jdk19

ChengJin01 · 2022-06-17T20:21:43Z

The test failures on MacOS/Aarch64 at https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_mac_Personal/60/tapResults/
e.g.

FAILED: test_vprintfFromDefaultLibWithVaList
java.util.NoSuchElementException: No value present
	at java.base/java.util.Optional.get(Optional.java:143)
	at org.openj9.test.jep389.valist.VaListTests.test_vprintfFromDefaultLibWithVaList(VaListTests.java:125)

was caused by the default library loading issue already detected previously on macOS in #14487 and was resolved with our own fix in OpenJDK at https://github.com/ibmruntimes/openj9-openjdk-jdk17/pull/112/files#diff-9658a8d1507ae5132a614d4c38ed6589ebab5bed19c3d03f726585aeb49dce29, https://github.com/ibmruntimes/openj9-openjdk-jdk18/pull/57/files#diff-9658a8d1507ae5132a614d4c38ed6589ebab5bed19c3d03f726585aeb49dce29, https://github.com/ibmruntimes/openj9-openjdk-jdk19/pull/5/files#diff-9658a8d1507ae5132a614d4c38ed6589ebab5bed19c3d03f726585aeb49dce29 and https://github.com/ibmruntimes/openj9-openjdk-jdk/pull/457/files#diff-9658a8d1507ae5132a614d4c38ed6589ebab5bed19c3d03f726585aeb49dce29 (not yet merged).

In addition, the tests on JDK19 are invalid given there is no our code in OpenJDK (the FFI specific code was totally removed in JDKnext due to the latest APIs in Java19) and all OpenJDK & OpenJ9 specific code are still under review. So I'll disable all test suites on all supported platforms in playlist.xml for the moment in case any partially merged code mess them up and re-enable them all after all code listed in #15068 for Java17/18/19 are merged if this PR needs to be merged individually.

tajila · 2022-07-12T14:07:56Z

jenkins test sanity plinux jdk19

tajila · 2022-07-12T14:08:08Z

jenkins test sanity amac jdk17

ChengJin01 · 2022-07-27T20:04:14Z

I just updated the test cases based on the latest changes in OpenJDK and added a couple of tests requested by JIT (working on thunk code on zLinux). I also removed all Valist specific test cases as they are outdated at this point and will be totally replaced with new tests once we have Valist implemented on Power in OpenJDK.

knn-k · 2022-08-18T06:00:58Z

runtime/tests/clinkerffi/CMakeLists.txt

@@ -23,6 +23,8 @@
 set(OMR_ENHANCED_WARNINGS OFF)
 j9vm_add_library(clinkerffitests SHARED
 	downcall.c
+	upcall.c
+	valist.c


There is no file named valist.c in this directory, isn't it?

Yes. The test cases for VaList are not ready and will be added once we have VaList support on Power & zLinux which I have been working on. So I just removed valist.c for the moment.

ChengJin01 · 2022-09-08T03:15:49Z

I just updated part of tests in UpcallMHWithStructTests.java for JDK17/18/19+ to support struct specific upcall on Linux/s390x given the padding in these tests is not required on Linux/s390x as explained in the PR intended for thunk generation code at #15596 (comment).

ChengJin01 · 2022-09-08T04:15:01Z

I just updated this PR to add the VaList specific test cases in JDK19+ which is intended for the VaList changes at ibmruntimes/openj9-openjdk-jdk#488 and ibmruntimes/openj9-openjdk-jdk19#27.

0xdaryl · 2022-09-20T12:31:43Z

...ctional/Java17andUp/src_170/org/openj9/test/jep389/upcall/UpcallMHWithMixedSigStruTests.java

+	}
+
+	@Test
+	public void test_addLongAndLong2FloatsFromStructByUpcallMH() throws Throwable {


This test fails on Windows for both HotSpot and OpenJ9. However, the problem seems to be on the "downcall" part of the test as the elements of the struct don't even reach the C function intact (i.e., before we can even do an upcall).

For HotSpot, printing the args reaching the C function (addLongAndLong2FloatsFromStructByUpcallMH) before the upcall:

IN C: addLongAndLong2FloatsFromStruct : arg1=555555555555, arg2.elem1=388697201, arg2.elem2=0.000000, arg2.elem3=11.250000

For OpenJ9, printing the args reaching the C function before the upcall:

IN C: addLongAndLong2FloatsFromStruct : arg1=555555555555, arg2.elem1=388697201, arg2.elem2=0.000000, arg2.elem3=11.250000

@0xdaryl, could you help double-check values[1] or ffiArgs[1] for the struct the code of inlInternalDowncallHandlerInvokeNative in downcall at

openj9/runtime/vm/BytecodeInterpreter.hpp

Line 5103 in b9b7339

values[i] = (void *)(U_64)ffiArgs[i];

?
I'd like to know whether or not the passed-in value is correct given the test passes on other platforms.

If the passed-in struct from inlInternalDowncallHandlerInvokeNative is correct but it gets corrupted in the C native function, the problem should originate from libffi on Windows in which case we should disable this test for the moment and will need to create an issue at https://github.com/libffi/libffi/issues or https://sourceware.org/mailman/listinfo/libffi-discuss to request them to fix it up.

I suspect the problem was caused by the wrong element type defined for elem1 in stru_Long_2_Floats at downcall.h

typedef struct stru_Long_2_Floats { long elem1; <--------- float elem2; float elem3; } stru_Long_2_Floats;

in which long is 4 bytes on Windows, which should be replaced with LONG (int64_t). I will update this definition to verify again on Windows to see how it goes.

I manually compile a JDK17 build on Windows and verified locally with the updated definition of stru_Long_2_Floats in /runtime/tests/clinkerffi/downcall.h

typedef struct stru_Long_2_Floats { LONG elem1; <------- float elem2; float elem3; } stru_Long_2_Floats;

and the test test_addLongAndLong2FloatsFromStructByUpcallMH passed on both OpenJ9 and Hotspot on Windows:

$ java -version openjdk version "17.0.5-internal" 2022-10-18 OpenJDK Runtime Environment (build 17.0.5-internal+0-adhoc.Administrator.openj9-openjdk-jdk17) Eclipse OpenJ9 VM (build jep389_419_ffi_upcall_v4_struct_v2_outofline_mh_v9_fin_v8_5-c3c03e474, JRE 17 Windows 10 amd64-64-Bit Compressed References 20220920_000000 (JIT enabled, AOT enabled ) OpenJ9 - c3c03e474 OMR - b1be89cf5 JCL - e988b7def79 based on jdk-17.0.5+5) $ java -version openjdk version "17.0.2" 2022-01-18 OpenJDK Runtime Environment (build 17.0.2+8-86) OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing) ... PASSED: test_addLongAndLong2FloatsFromStructByUpcallMH

@0xdaryl, please double-check the test on your side (note that the test library clinkerffitests.dll needs to be regenerated with the latest downcall.h in this PR)

0xdaryl · 2022-09-20T12:38:32Z

test/functional/Java19andUp/src/org/openj9/test/jep424/upcall/UpcallMHWithStructTests.java

+public class UpcallMHWithStructTests {
+	private static String osName = System.getProperty("os.name").toLowerCase();
+	private static String arch = System.getProperty("os.arch").toLowerCase();
+	private static boolean isLinuxS390x = osName.contains("linux") && arch.equals("s390x");


Windows does not require the structure padding either, so all places that check for isLinuxS390x will have to check for Windows as well.

I wonder if it would be better to introduce a requiresStructurePadding boolean here that is initialized to false for Z and Windows rather than adding OS checks for many of the tests. It seems a bit cleaner in my opinion.

Same for all other versions of UpcallMHWithStructTests.java.

I will update the tests on Java17/18/19 based against the suggestion above.

@0xdaryl, I've updated (and verified locally) the tests in Java17/18/19 by replacing the OS check with isStructPaddingNotRequired for zLinux & Win so as to minimize the changes. Please help check to see whether it works good with your thunk code on Windows.

tajila · 2022-10-03T18:08:52Z

@ChengJin01 is testing disabled for x86?

ChengJin01 · 2022-10-03T18:19:17Z

@ChengJin01 is testing disabled for x86?

Yes. All FFI related test suites are disabled on x86 in JDK17/18/19+ for the moment
https://github.com/eclipse-openj9/openj9/blob/5b9e56396c9e0ac2bc2f2a35884cb29558f1a8d6/test/functional/Java17andUp/playlist.xml
https://github.com/eclipse-openj9/openj9/blob/5b9e56396c9e0ac2bc2f2a35884cb29558f1a8d6/test/functional/Java18andUp/playlist.xml
https://github.com/eclipse-openj9/openj9/blob/5b9e56396c9e0ac2bc2f2a35884cb29558f1a8d6/test/functional/Java19andUp/playlist.xml
given the thunk generation code on x86 at #15926 has not yet merged (still under code review).

tajila · 2022-10-03T19:50:36Z

jenkins test sanity,extended plinux,zlinux,aix,amac jdk19

ChengJin01 · 2022-10-03T20:12:36Z

The downcall & upcall related test suites are enabled in JDK17/18/19+ on AIX & pLinux, zLinux and Aarch64 and VaList tests are only enabled on AIX & pLinux in JDK19+.

tajila · 2022-10-04T04:10:14Z

jenkins test sanity,extended plinux,zlinux,aix,amac jdk19

ChengJin01 · 2022-10-04T04:31:51Z

I just updated a couple of latest native test code for VaList which were missing in this PR.

The changes include all test suites intended for downcall and upcall in Java17, 18 and 19 which can be enabled on the following platforms: 1)downcall: all supported platforms 2)upcall: AIX/PPC and Linux/PPC64LE Signed-off-by: Cheng Jin <jincheng@ca.ibm.com>

tajila · 2022-10-04T13:03:34Z

jenkins test sanity,extended plinux jdk19

tajila · 2022-10-04T16:38:52Z

jenkins test sanity,extended zlinux,aix,amac jdk19

This was referenced Jun 13, 2022

Enable the downcall tests for JEP389 on JDK17 #14812

Closed

Enable the downcall tests for JEP419 on JDK18 and JDKnext #14811

Closed

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch from d90ca1e to 3082fa4 Compare June 13, 2022 18:51

ChengJin01 requested a review from tajila June 13, 2022 19:26

ChengJin01 mentioned this pull request Jun 13, 2022

Implement the functionalities of Foreign Linker API #12412

Closed

tajila approved these changes Jun 14, 2022

View reviewed changes

tajila requested a review from llxia June 14, 2022 13:43

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch from 3082fa4 to 82d08a7 Compare June 15, 2022 18:16

llxia approved these changes Jun 16, 2022

View reviewed changes

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch 3 times, most recently from 50228ae to 5df601c Compare June 21, 2022 23:16

ChengJin01 mentioned this pull request Jun 23, 2022

[Part 1/JDK19] FFI Upcall: creating the upcall stub (java part) #15307

Merged

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch from 5df601c to d39eb85 Compare July 5, 2022 16:13

ChengJin01 mentioned this pull request Jul 19, 2022

Several JEP424 fixes ibmruntimes/openj9-openjdk-jdk19#15

Merged

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch 4 times, most recently from 667e81c to a971b29 Compare July 27, 2022 20:00

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch 2 times, most recently from 0415284 to d975979 Compare August 17, 2022 19:37

knn-k reviewed Aug 18, 2022

View reviewed changes

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch from b181220 to b9c0f7f Compare September 8, 2022 03:10

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch 2 times, most recently from 9c355a7 to da9331b Compare September 8, 2022 04:12

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch 2 times, most recently from da64ca6 to 0a70e5e Compare September 14, 2022 22:39

0xdaryl reviewed Sep 20, 2022

View reviewed changes

0xdaryl mentioned this pull request Sep 20, 2022

X64 FFI upcall thunk generation #15926

Merged

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch 3 times, most recently from b6dfebe to 6f7ce5c Compare September 22, 2022 17:34

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch from 6f7ce5c to 5b9e563 Compare September 27, 2022 17:11

ChengJin01 mentioned this pull request Oct 3, 2022

[JEP424] Implement VaList on Power (Linux & AIX) in JDKnext ibmruntimes/openj9-openjdk-jdk#488

Merged

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch from 5b9e563 to 171327c Compare October 3, 2022 20:10

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch from 171327c to 35847f6 Compare October 4, 2022 04:30

ChengJin01 force-pushed the enable_ffi_downcall_upcall_tests_jdk17_18_19 branch from 35847f6 to c3b12b5 Compare October 4, 2022 04:34

tajila merged commit 8faf666 into eclipse-openj9:master Oct 5, 2022

pshipton added the comp:test label Nov 2, 2022

ChengJin01 mentioned this pull request Jan 30, 2023

jdk19 Jep424Tests_testLinkerFfi_UpCall_0 test_addBoolAndBoolsFromStructWithNestedBoolArrayByUpcallMH expected [true] but found [false] #16619

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable downcall & upcall test suites for Java 17/18/19 #15310

Enable downcall & upcall test suites for Java 17/18/19 #15310

ChengJin01 commented Jun 13, 2022

ChengJin01 commented Jun 13, 2022 •

edited

Loading

tajila commented Jun 14, 2022

llxia left a comment

tajila commented Jun 17, 2022

tajila commented Jun 17, 2022

ChengJin01 commented Jun 17, 2022 •

edited

Loading

tajila commented Jul 12, 2022

tajila commented Jul 12, 2022

ChengJin01 commented Jul 27, 2022 •

edited

Loading

knn-k Aug 18, 2022

ChengJin01 Aug 18, 2022

ChengJin01 commented Sep 8, 2022

ChengJin01 commented Sep 8, 2022

0xdaryl Sep 20, 2022

ChengJin01 Sep 20, 2022 •

edited

Loading

ChengJin01 Sep 20, 2022 •

edited

Loading

ChengJin01 Sep 21, 2022 •

edited

Loading

ChengJin01 Sep 21, 2022

0xdaryl Sep 20, 2022

ChengJin01 Sep 20, 2022 •

edited

Loading

ChengJin01 Sep 20, 2022 •

edited

Loading

tajila commented Oct 3, 2022

ChengJin01 commented Oct 3, 2022

tajila commented Oct 3, 2022

ChengJin01 commented Oct 3, 2022 •

edited

Loading

tajila commented Oct 4, 2022

ChengJin01 commented Oct 4, 2022 •

edited

Loading

tajila commented Oct 4, 2022

tajila commented Oct 4, 2022

Enable downcall & upcall test suites for Java 17/18/19 #15310

Enable downcall & upcall test suites for Java 17/18/19 #15310

Conversation

ChengJin01 commented Jun 13, 2022

ChengJin01 commented Jun 13, 2022 • edited Loading

tajila commented Jun 14, 2022

llxia left a comment

Choose a reason for hiding this comment

tajila commented Jun 17, 2022

tajila commented Jun 17, 2022

ChengJin01 commented Jun 17, 2022 • edited Loading

tajila commented Jul 12, 2022

tajila commented Jul 12, 2022

ChengJin01 commented Jul 27, 2022 • edited Loading

knn-k Aug 18, 2022

Choose a reason for hiding this comment

ChengJin01 Aug 18, 2022

Choose a reason for hiding this comment

ChengJin01 commented Sep 8, 2022

ChengJin01 commented Sep 8, 2022

0xdaryl Sep 20, 2022

Choose a reason for hiding this comment

ChengJin01 Sep 20, 2022 • edited Loading

Choose a reason for hiding this comment

ChengJin01 Sep 20, 2022 • edited Loading

Choose a reason for hiding this comment

ChengJin01 Sep 21, 2022 • edited Loading

Choose a reason for hiding this comment

ChengJin01 Sep 21, 2022

Choose a reason for hiding this comment

0xdaryl Sep 20, 2022

Choose a reason for hiding this comment

ChengJin01 Sep 20, 2022 • edited Loading

Choose a reason for hiding this comment

ChengJin01 Sep 20, 2022 • edited Loading

Choose a reason for hiding this comment

tajila commented Oct 3, 2022

ChengJin01 commented Oct 3, 2022

tajila commented Oct 3, 2022

ChengJin01 commented Oct 3, 2022 • edited Loading

tajila commented Oct 4, 2022

ChengJin01 commented Oct 4, 2022 • edited Loading

tajila commented Oct 4, 2022

tajila commented Oct 4, 2022

ChengJin01 commented Jun 13, 2022 •

edited

Loading

ChengJin01 commented Jun 17, 2022 •

edited

Loading

ChengJin01 commented Jul 27, 2022 •

edited

Loading

ChengJin01 Sep 20, 2022 •

edited

Loading

ChengJin01 Sep 20, 2022 •

edited

Loading

ChengJin01 Sep 21, 2022 •

edited

Loading

ChengJin01 Sep 20, 2022 •

edited

Loading

ChengJin01 Sep 20, 2022 •

edited

Loading

ChengJin01 commented Oct 3, 2022 •

edited

Loading

ChengJin01 commented Oct 4, 2022 •

edited

Loading