8274912: Eagerly generate native invokers #593

JornVernee · 2021-10-07T14:29:47Z

Hi,

This patch is a refactoring of VM code pertaining to downcalls. As outlined in the JBS issue, this patch makes 'native invoker' generation currently done by C2 eager.

Most of the changes in this patch are moving around existing code:

The native invoker generation code is moved from sharedRuntime*.cpp to universalNativeInvoker*.cpp, to match what is done for upcalls.
C2 & nmethod scaffolding for registering native invokers has been removed (lifetime is now managed in Java code).
Now, C2 will only intrinsify trivial downcalls, which means some of the code was simplified there.
To reduce code duplication, several utilities have been added:
- ArgumentShuffle (and impls of CallConvClosure): a utility class for computing argument shuffles and generating shuffling code. The shuffling code is heavily based on what is done today for optimized upcalls on x64.
- To support the above, I've made a copy of the ComputeMoveOrder utility from SharedRuntime code (which was only implemented on x64), and adapted it to our use case.
- RegSpillFill: similarly, a utility for computing and generating register spill and fill code (given a list of registers).
- oop_cast: this was a pre-existing utility that was renamed (from 'cast') and moved into a separate file.
The linkToNative method handle stub now just loads the invoker address from the appendix argument and jumps to it, instead of jumping to the fallback method handle.

The implementation passes all the jdk_foreign tests on Windows/x64, Linux/x64 (WSL), and Linux/AArch64. Though, there were some pre-exsiting test failures on AArch64 wrt variadic functions.

Thanks,
Jorn

Progress

Change must not contain extraneous whitespace
Change must be properly reviewed

Issue

JDK-8274912: Eagerly generate native invokers

Reviewers

Nick Gasson (@nick-arm - no project role) ⚠️ Review applies to 493165e
Maurizio Cimadamore (@mcimadamore - Committer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/panama-foreign pull/593/head:pull/593
$ git checkout pull/593

Update a local copy of the PR:
$ git checkout pull/593
$ git pull https://git.openjdk.java.net/panama-foreign pull/593/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 593

View PR using the GUI difftool:
$ git pr show -t 593

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/panama-foreign/pull/593.diff

- add fp to gp move - redo frame allocation calc - add stack argument offset to account for shadow space

- Fix pre-existing bug in register shuffle code - Turn on all UpcallTest types.

JornVernee · 2021-10-07T14:30:50Z

test/jdk/java/foreign/TestUpcall.java

+/*
+ * @test id=scope
+ * @requires ((os.arch == "amd64" | os.arch == "x86_64") & sun.arch.data.model == "64") | os.arch == "aarch64"
+ * @modules jdk.incubator.foreign/jdk.internal.foreign
+ * @build NativeTestHelper CallGeneratorHelper TestUpcall
+ *
+ * @run testng/othervm -XX:+IgnoreUnrecognizedVMOptions -XX:-VerifyDependencies
+ *   --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17
+ *   -DUPCALL_TEST_TYPE=SCOPE
+ *   TestUpcall
+ */
+
+/*
+ * @test id=no_scope
+ * @requires ((os.arch == "amd64" | os.arch == "x86_64") & sun.arch.data.model == "64") | os.arch == "aarch64"
+ * @modules jdk.incubator.foreign/jdk.internal.foreign
+ * @build NativeTestHelper CallGeneratorHelper TestUpcall
+ *
+ * @run testng/othervm -XX:+IgnoreUnrecognizedVMOptions -XX:-VerifyDependencies
+ *   --enable-native-access=ALL-UNNAMED -Dgenerator.sample.factor=17
+ *   -DUPCALL_TEST_TYPE=NO_SCOPE
+ *   TestUpcall
+ */
+


It seems that not all the upcall test configurations were being run, so I've added them back here. (these are separate jtreg tests, so shouldn't run into any problems with timeouts).

bridgekeeper · 2021-10-07T14:31:17Z

👋 Welcome back jvernee! A progress list of the required criteria for merging this PR into foreign-memaccess+abi will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

JornVernee · 2021-10-07T14:31:46Z

src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp

+    else if (rOop == j_rarg1)
+      oop_slot = 1;
+    else if (rOop == j_rarg2)
+      oop_slot = 2;
+    else if (rOop == j_rarg3)
+      oop_slot = 3;
+    else if (rOop == j_rarg4)
+      oop_slot = 4;
+    else if (rOop == j_rarg5)
+      oop_slot = 5;
+    else if (rOop == j_rarg6)
+      oop_slot = 6;
+    else {
+      assert(rOop == j_rarg7, "wrong register");
+      oop_slot = 7;
+    }
+
+    oop_slot = oop_slot * VMRegImpl::slots_per_word + oop_handle_offset;
+    int offset = oop_slot*VMRegImpl::stack_slot_size;
+
+    map->set_oop(VMRegImpl::stack2reg(oop_slot));
+    // Store oop in handle area, may be NULL
+    str(rOop, Address(sp, offset));
+    if (is_receiver) {
+      *receiver_offset = offset;
+    }
+
+    cmp(rOop, zr);
+    lea(rHandle, Address(sp, offset));
+    // conditionally move a NULL
+    csel(rHandle, zr, rHandle, Assembler::EQ);
+  }
+
+  // If arg is on the stack then place it otherwise it is already in correct reg.
+  if (dst.first()->is_stack()) {
+    str(rHandle, Address(sp, reg2offset_out(dst.first())));
+  }
+}
+
+// A float arg may have to do float reg int reg conversion
+void MacroAssembler::float_move(VMRegPair src, VMRegPair dst) {
+  assert(src.first()->is_stack() && dst.first()->is_stack() ||
+         src.first()->is_reg() && dst.first()->is_reg(), "Unexpected error");
+  if (src.first()->is_stack()) {
+    if (dst.first()->is_stack()) {
+      ldrw(rscratch1, Address(rfp, reg2offset_in(src.first())));
+      strw(rscratch1, Address(sp, reg2offset_out(dst.first())));
+    } else {
+      ShouldNotReachHere();
+    }
+  } else if (src.first() != dst.first()) {
+    if (src.is_single_phys_reg() && dst.is_single_phys_reg())
+      fmovs(dst.first()->as_FloatRegister(), src.first()->as_FloatRegister());
+    else
+      ShouldNotReachHere();
+  }
+}
+
+// A long move
+void MacroAssembler::long_move(VMRegPair src, VMRegPair dst) {
+  if (src.first()->is_stack()) {
+    if (dst.first()->is_stack()) {
+      // stack to stack
+      ldr(rscratch1, Address(rfp, reg2offset_in(src.first())));
+      str(rscratch1, Address(sp, reg2offset_out(dst.first())));
+    } else {
+      // stack to reg
+      ldr(dst.first()->as_Register(), Address(rfp, reg2offset_in(src.first())));
+    }
+  } else if (dst.first()->is_stack()) {
+    // reg to stack
+    // Do we really have to sign extend???
+    // __ movslq(src.first()->as_Register(), src.first()->as_Register());
+    str(src.first()->as_Register(), Address(sp, reg2offset_out(dst.first())));
+  } else {
+    if (dst.first() != src.first()) {
+      mov(dst.first()->as_Register(), src.first()->as_Register());
+    }
+  }
+}
+
+
+// A double move
+void MacroAssembler::double_move(VMRegPair src, VMRegPair dst) {
+  assert(src.first()->is_stack() && dst.first()->is_stack() ||
+         src.first()->is_reg() && dst.first()->is_reg(), "Unexpected error");
+  if (src.first()->is_stack()) {
+    if (dst.first()->is_stack()) {
+      ldr(rscratch1, Address(rfp, reg2offset_in(src.first())));
+      str(rscratch1, Address(sp, reg2offset_out(dst.first())));
+    } else {
+      ShouldNotReachHere();
+    }
+  } else if (src.first() != dst.first()) {
+    if (src.is_single_phys_reg() && dst.is_single_phys_reg())
+      fmovd(dst.first()->as_FloatRegister(), src.first()->as_FloatRegister());
+    else
+      ShouldNotReachHere();
+  }
+}


Similar to what was done for x64, I've moved the aarch64 shuffling helpers to macroassembler

JornVernee · 2021-10-07T14:32:40Z

src/hotspot/cpu/x86/macroAssembler_x86.cpp

      movq(Address(rsp, reg2offset_out(dst.first())), src.first()->as_Register());
    }
  } else if (dst.is_single_phys_reg()) {
    assert(src.is_single_reg(),  "not a stack pair");
-    movq(dst.first()->as_Register(), Address(rbp, reg2offset_out(src.first())));
+    movq(dst.first()->as_Register(), Address(rbp, reg2offset_in(src.first())));


A pre-existing bug found when testing (same below). Loads from rbp should always be paired with reg2offset_in

mlbridge · 2021-10-07T14:43:41Z

Webrevs

mcimadamore · 2021-10-07T15:06:32Z

src/hotspot/cpu/arm/universalNativeInvoker_arm.cpp

@@ -29,3 +29,17 @@ address ProgrammableInvoker::generate_adapter(jobject jabi, jobject jlayout) {
  Unimplemented();
  return nullptr;
 }
+
+RuntimeStub* ProgrammableInvoker::make_native_invoker(BasicType* signature,


Maybe crazy idea - can we make it so that, by default make_native_invoker will just throw Unimplemented() in the ProgrammableInvoker class - and if clients want to provide implementation for it, they have to subclass and define an implementation? This might avoid need for stubbing things out.

Well, I guess in that case each platform would still need to define a factory method for the platform specific impl, so that platform agnostic code can reach it, and that would then need to be stubbed on unsupported platforms.

But, either way, I think it's better to stick with the conventions used in other VM code, which is to stub things out.

JornVernee · 2021-10-07T15:33:20Z

@nick-arm if you want to, could you take a look at the AArch64 bits?

nick-arm · 2021-10-11T05:07:19Z

@nick-arm if you want to, could you take a look at the AArch64 bits?

Sure, will do.

nick-arm

AArch64 changes look fine, just a couple of trivial naming comments. I'm also seeing the varargs test failure now on the foreign-memaccess+abi branch.

src/hotspot/cpu/aarch64/universalNativeInvoker_aarch64.cpp

src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp

JornVernee · 2021-10-11T10:17:25Z

@nick-arm Thanks for the good suggestions, I've applied them.

nick-arm · 2021-10-13T07:12:49Z

The implementation passes all the jdk_foreign tests on Windows/x64, Linux/x64 (WSL), and Linux/AArch64. Though, there were some pre-exsiting test failures on AArch64 wrt variadic functions.

Looks like the varargs failures are caused by this change in #576:

https://github.com/openjdk/panama-foreign/pull/576/files#diff-9fbf245dc20c824d5e9f3b1864a18284f110ae808cf11782457f8462e8ad8ce0L211

The STACK_VARARGS_ATTRIBUTE_NAME attribute used to be set only on macOS and was used to force all variadic arguments onto the stack, but that now happens unconditionally on Linux as well.

JornVernee · 2021-10-14T12:41:53Z

The STACK_VARARGS_ATTRIBUTE_NAME attribute used to be set only on macOS and was used to force all variadic arguments onto the stack, but that now happens unconditionally on Linux as well.

Ah, nice find!

I think we can just add a check to see if we are on the Mac ABI and only then do the adjustment for varargs?

JornVernee · 2021-10-15T12:00:22Z

I've filed an issue here: https://bugs.openjdk.java.net/browse/JDK-8275332

Rather than just checking whether we are running on MacOS before adjusting for var args, I think it would be better to have it as a configuration flag passed to CallArranger, or perhaps to split the AArch64 CallArranger into a base class and 2 sub-classes for each platform, that way we can add CallArranger tests for each platform that will still run on all platforms.

mcimadamore

Looks very good.

openjdk · 2021-10-15T13:35:18Z

@JornVernee This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8274912: Eagerly generate native invokers

Reviewed-by: ngasson, mcimadamore

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 266 new commits pushed to the foreign-memaccess+abi branch:

e5e373e: Automatic merge of master into foreign-memaccess+abi
43a0653: Automatic merge of jdk:master into master
c0f3e1d: 8271071: accessibility of a table on macOS lacks cell navigation
4cb7124: 8262912: ciReplay: replay does not simulate unresolved classes
322b130: 8275106: Cleanup Iterator usages in java.desktop
c355704: 8041125: ColorConvertOp filter much slower in JDK 8 compared to JDK7
9623d5b: 8275265: java/nio/channels tests needing large heap sizes fail on x86_32
21df412: 8275240: Change nested classes in jdk.attach to static nested classes
a16f2d0: 8272908: Missing coverage for certain classes in com.sun.org.apache.xml.internal.security
ede3f4e: 8274795: AArch64: avoid spilling and restoring r18 in macro assembler
... and 256 more: https://git.openjdk.java.net/panama-foreign/compare/1414f2c2c56409ee8b7a51026f28a1b5277c03f3...foreign-memaccess+abi

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the foreign-memaccess+abi branch, type /integrate in a new comment.

JornVernee · 2021-10-15T14:11:11Z

/integrate

openjdk · 2021-10-15T14:12:56Z

Going to push as commit 1aa1c29.
Since your change was applied there have been 266 commits pushed to the foreign-memaccess+abi branch:

e5e373e: Automatic merge of master into foreign-memaccess+abi
43a0653: Automatic merge of jdk:master into master
c0f3e1d: 8271071: accessibility of a table on macOS lacks cell navigation
4cb7124: 8262912: ciReplay: replay does not simulate unresolved classes
322b130: 8275106: Cleanup Iterator usages in java.desktop
c355704: 8041125: ColorConvertOp filter much slower in JDK 8 compared to JDK7
9623d5b: 8275265: java/nio/channels tests needing large heap sizes fail on x86_32
21df412: 8275240: Change nested classes in jdk.attach to static nested classes
a16f2d0: 8272908: Missing coverage for certain classes in com.sun.org.apache.xml.internal.security
ede3f4e: 8274795: AArch64: avoid spilling and restoring r18 in macro assembler
... and 256 more: https://git.openjdk.java.net/panama-foreign/compare/1414f2c2c56409ee8b7a51026f28a1b5277c03f3...foreign-memaccess+abi

Your commit was automatically rebased without conflicts.

openjdk · 2021-10-15T14:13:15Z

@JornVernee Pushed as commit 1aa1c29.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

JornVernee added 27 commits September 24, 2021 14:52

WIP - shuffle arguments in native invoker

6d633ac

WIP - added shuffle + enabled stack args

1077a03

WIP

31f2483

- add fp to gp move - redo frame allocation calc - add stack argument offset to account for shadow space

WIP - move invoker generator to universalNativeInvoker_x86_64.cpp

92a3f92

WIP - eagerly generate native invoker

234b315

WIP - add invoker

df9d0e1

WIP - remove native invoker compilation infra

771b91e

WIP - Clean up native invoker code from SharedRuntime

4f1ab27

Misc cleanups + RegSpillFill util

aaea52f

WIP - Add ArgumentShuffle util

8946f0e

Rebase Fixes

c334756

Add oopCast.inline.hpp

b345f1a

Linux fixes

5173749

Remove stack arg bits for now

142e973

Add missing stubs + remove more stack arg bits

fe304ca

patch up method handle stubs on all platforms

4d9f6e5

Add missing jump_to_native_invoker decls

ac8e99f

Add missing AArch64 include

5f79ac4

WIP - native invoker support for aarch64

4ec5bc8

WIP - attempt at aarch64 native invoker impl

24b9b11

Trim whitespace

2ce6431

Add aarch64 linkToNative impl + remove redundant blob decode

9764c83

Dissable aarch64 invoker for now

1bd9d88

Fix scratch register conflicts for AArch64

af3ca8a

simplifiy ForeignCMO

8bbbb87

- Simplify xmm to gp reg move code

709a91f

- Fix pre-existing bug in register shuffle code - Turn on all UpcallTest types.

Last AArch64 fixes

8e9e4a5

JornVernee commented Oct 7, 2021

View reviewed changes

Whitespace fix

493165e

openjdk bot added the rfr Ready for review label Oct 7, 2021

mcimadamore reviewed Oct 7, 2021

View reviewed changes

nick-arm approved these changes Oct 11, 2021

View reviewed changes

src/hotspot/cpu/aarch64/universalNativeInvoker_aarch64.cpp Outdated Show resolved Hide resolved

src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp Outdated Show resolved Hide resolved

AArch64 comments

febe2ff

Address offline review comments

fdae163

mcimadamore approved these changes Oct 15, 2021

View reviewed changes

openjdk bot added the ready Ready to be integrated label Oct 15, 2021

openjdk bot closed this Oct 15, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Ready to be integrated rfr Ready for review labels Oct 15, 2021

JornVernee deleted the Eager_Invoker_RB branch November 2, 2021 15:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8274912: Eagerly generate native invokers #593

8274912: Eagerly generate native invokers #593

JornVernee commented Oct 7, 2021 •

edited by openjdk bot

JornVernee Oct 7, 2021

bridgekeeper bot commented Oct 7, 2021

JornVernee Oct 7, 2021 •

edited

JornVernee Oct 7, 2021

mlbridge bot commented Oct 7, 2021 •

edited

mcimadamore Oct 7, 2021

JornVernee Oct 7, 2021

JornVernee commented Oct 7, 2021

nick-arm commented Oct 11, 2021

nick-arm left a comment

JornVernee commented Oct 11, 2021

nick-arm commented Oct 13, 2021

JornVernee commented Oct 14, 2021

JornVernee commented Oct 15, 2021 •

edited

mcimadamore left a comment

openjdk bot commented Oct 15, 2021 •

edited

JornVernee commented Oct 15, 2021

openjdk bot commented Oct 15, 2021

openjdk bot commented Oct 15, 2021

8274912: Eagerly generate native invokers #593

8274912: Eagerly generate native invokers #593

Conversation

JornVernee commented Oct 7, 2021 • edited by openjdk bot

Progress

Issue

Reviewers

Reviewing

JornVernee Oct 7, 2021

Choose a reason for hiding this comment

bridgekeeper bot commented Oct 7, 2021

JornVernee Oct 7, 2021 • edited

Choose a reason for hiding this comment

JornVernee Oct 7, 2021

Choose a reason for hiding this comment

mlbridge bot commented Oct 7, 2021 • edited

Webrevs

mcimadamore Oct 7, 2021

Choose a reason for hiding this comment

JornVernee Oct 7, 2021

Choose a reason for hiding this comment

JornVernee commented Oct 7, 2021

nick-arm commented Oct 11, 2021

nick-arm left a comment

Choose a reason for hiding this comment

JornVernee commented Oct 11, 2021

nick-arm commented Oct 13, 2021

JornVernee commented Oct 14, 2021

JornVernee commented Oct 15, 2021 • edited

mcimadamore left a comment

Choose a reason for hiding this comment

openjdk bot commented Oct 15, 2021 • edited

JornVernee commented Oct 15, 2021

openjdk bot commented Oct 15, 2021

openjdk bot commented Oct 15, 2021

JornVernee commented Oct 7, 2021 •

edited by openjdk bot

JornVernee Oct 7, 2021 •

edited

mlbridge bot commented Oct 7, 2021 •

edited

JornVernee commented Oct 15, 2021 •

edited

openjdk bot commented Oct 15, 2021 •

edited