8312522: Implementation of Foreign Function & Memory API #15103

JornVernee · 2023-08-01T10:29:06Z

This patch contains the implementation of the foreign linker & memory API JEP for Java 22. The initial patch is composed of commits brought over directly from the panama-foreign repo. The main changes found in this patch come from the following PRs:

8308858: FFM API and strings panama-foreign#836 Where previous iterations supported converting Java strings to and from native strings in the UTF-8 encoding, we've extended the supported encoding to all the encodings found in the java.nio.charset.StandardCharsets class.
8310048: Drop sequence layout factory with inferred count panama-foreign#838 We dropped the MemoryLayout::sequenceLayout factory method which inferred the size of the sequence to be Long.MAX_VALUE, as this led to confusion among clients. A client is now required to explicitly specify the sequence size.
8308293: A linker should expose the layouts it supports panama-foreign#839 A new API was added: Linker::canonicalLayouts, which exposes a map containing the platform-specific mappings of common C type names to memory layouts.
8310362: Improve composability of handle derived from layouts panama-foreign#840 Memory access varhandles, as well as byte offset and slice handles derived from memory layouts, now feature an additional 'base offset' coordinate that is added to the offset computed by the handle. This allows composing these handles with other offset computation strategies that may not be based on the same memory layout. This addresses use-cases where clients are working with 'dynamic' layouts, whose size might not be known statically, such as variable length arrays, or variable size matrices.
8310820: Remove MemorySegment::segmentOffset panama-foreign#841 Remove this now redundant API. Clients can simply use the difference between the base address of two memory segments.
8311533: SegmentAllocator::allocateArray call can be ambiguous panama-foreign#845 Disambiguate uses of SegmentAllocator::allocateArray, by renaming methods that both allocate + initialize memory segments to allocateFrom. (see the original PR for the problematic case)
8312059: Clarify the documention for variadic functions panama-foreign#846 Improve the documentation for variadic functions.
8312615: Ensure jdk_foreign tests pass on linux-x86 panama-foreign#849 Several test fixes to make sure the jdk_foreign tests can pass on 32-bit machines, taking linux-x86 as a test bed.
8312981: Make the linker API required panama-foreign#850 Make the linker API required. The Linker::nativeLinker method is not longer allowed to throw an UnsupportedOperationException on unsupported platforms. All tests are turned on by default, instead of being skipped when the linker is not present.
8313005: Ensure native access check can fold away panama-foreign#851 Minor code tweaks to ensure the JIT can constant fold through native access checks if the accessing class is statically known (see commit/original PR for changes).
8313265: Move the FFM API out of preview panama-foreign#853 Remove all the @PreviewFeature annotations from the API. Update all @since tags in the Javadoc to @since 22 (per JEP 12). Update tests and benchmarks to no longer build and run using --enable-preview (or the @enablePreview jtreg tag).

I want to call out in particular that this patch finalizes the FFM API (by moving it out of preview), and requires all JDK implementations to implement it. Most ports already have full FFM API support. The ones that are missing are: s390 (currently under review), windows-x86 (deprecated for removal), and linux-86 & arm32 (which can both be implemented using the fallback linker 1 2).

Testing: tier 1-5

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue
Change requires a JEP request to be targeted
Change requires CSR request JDK-8312523 to be approved

Issues

JDK-8312522: Implementation of Foreign Function & Memory API (Enhancement - P3)
JDK-8310626: JEP 454: Foreign Function & Memory API (JEP)
JDK-8312523: Implementation of Foreign Function & Memory API (CSR)

Reviewers

David Holmes (@dholmes-ora - Reviewer) ⚠️ Review applies to b0a1abaf
Paul Sandoz (@PaulSandoz - Reviewer) ⚠️ Review applies to 0244845a
Maurizio Cimadamore (@mcimadamore - Reviewer) ⚠️ Review applies to 1f3248df
Alan Bateman (@AlanBateman - Reviewer) ⚠️ Review applies to 2bc0a650

Contributors

Maurizio Cimadamore <mcimadamore@openjdk.org>
Jorn Vernee <jvernee@openjdk.org>
Per Minborg <pminborg@openjdk.org>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/15103/head:pull/15103
$ git checkout pull/15103

Update a local copy of the PR:
$ git checkout pull/15103
$ git pull https://git.openjdk.org/jdk.git pull/15103/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 15103

View PR using the GUI difftool:
$ git pr show -t 15103

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/15103.diff

Webrev

Link to Webrev Comment

Reviewed-by: jvernee

Reviewed-by: mcimadamore

Reviewed-by: jvernee

Reviewed-by: mcimadamore

Reviewed-by: jvernee

8310646: Javadoc around prototype-less functions might be incorrect Reviewed-by: mcimadamore

Reviewed-by: mcimadamore

bridgekeeper · 2023-08-01T10:29:48Z

👋 Welcome back jvernee! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2023-08-01T10:31:10Z

@JornVernee The following labels will be automatically applied to this pull request:

build
core-libs
hotspot
nio
shenandoah

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

JornVernee · 2023-08-01T10:33:59Z

/label remove shenandoah

JornVernee · 2023-08-01T10:34:45Z

/label add hotspot-compiler

openjdk · 2023-08-01T10:35:23Z

@JornVernee
The shenandoah label was successfully removed.

openjdk · 2023-08-01T10:36:10Z

@JornVernee
The hotspot-compiler label was successfully added.

JornVernee · 2023-08-01T10:36:37Z

/contributor add @mcimadamore
/contributor add @JornVernee

openjdk · 2023-08-01T10:36:54Z

@JornVernee
Contributor Maurizio Cimadamore <mcimadamore@openjdk.org> successfully added.

JornVernee · 2023-08-01T10:37:36Z

/jep JDK-8310626

ExE-Boss · 2023-09-30T15:50:03Z

Maybe include openjdk/panama-foreign#864?

JornVernee · 2023-09-30T16:19:53Z

Maybe include openjdk/panama-foreign#864?

The leftover implementation-only changes will be ported over separately.

JornVernee · 2023-10-06T16:10:54Z

I've addressed a test failure in b4a7b7a that was found after merging this patch with the most recent master branch.

The test in questions checks for linter warnings for restricted methods. It depends on the FFM API for that, which is still preview in the master branch, but no longer preview as part of this patch. Hence the warnings the compiler outputs are different, and the test fails. I've fixed to test. The preview flags are no longer needed.

openjdk · 2023-10-12T17:52:04Z

@JornVernee This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8312522: Implementation of Foreign Function & Memory API

Co-authored-by: Maurizio Cimadamore <mcimadamore@openjdk.org>
Co-authored-by: Jorn Vernee <jvernee@openjdk.org>
Co-authored-by: Per Minborg <pminborg@openjdk.org>
Reviewed-by: dholmes, psandoz, mcimadamore, alanb

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 67 new commits pushed to the master branch:

9728e21: 8317967: Enhance test/jdk/javax/net/ssl/TLSCommon/SSLEngineTestCase.java to handle default cases
a8473b7: 8317336: Assertion error thrown during 'this' escape analysis
16fd43c: 8316693: Simplify at-requires checkDockerSupport()
c1f698d: 8317956: Make jdk.internal.util.Architecture current architecture final
9355431: 8316426: Optimization for HexFormat.formatHex
32ccf01: 8317772: NMT: Make peak values available in release builds
4c79e7d: 8170817: G1: Returning MinTLABSize from unsafe_max_tlab_alloc causes TLAB flapping
7633a76: 8317998: Temporarily disable malformed control flow assert to reduce noise in testing
00ef9f9: 8316947: Write a test to check textArea triggers MouseEntered/MouseExited events properly
77dc891: 8317963: Serial: Remove unused GenerationIsInReservedClosure
... and 57 more: https://git.openjdk.org/jdk/compare/b3cc0c84316dd59f406a6fa23fcaf3d029910843...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

JornVernee · 2023-10-12T19:47:53Z

/integrate

openjdk · 2023-10-12T19:50:18Z

Going to push as commit 32ac72c.
Since your change was applied there have been 67 commits pushed to the master branch:

9728e21: 8317967: Enhance test/jdk/javax/net/ssl/TLSCommon/SSLEngineTestCase.java to handle default cases
a8473b7: 8317336: Assertion error thrown during 'this' escape analysis
16fd43c: 8316693: Simplify at-requires checkDockerSupport()
c1f698d: 8317956: Make jdk.internal.util.Architecture current architecture final
9355431: 8316426: Optimization for HexFormat.formatHex
32ccf01: 8317772: NMT: Make peak values available in release builds
4c79e7d: 8170817: G1: Returning MinTLABSize from unsafe_max_tlab_alloc causes TLAB flapping
7633a76: 8317998: Temporarily disable malformed control flow assert to reduce noise in testing
00ef9f9: 8316947: Write a test to check textArea triggers MouseEntered/MouseExited events properly
77dc891: 8317963: Serial: Remove unused GenerationIsInReservedClosure
... and 57 more: https://git.openjdk.org/jdk/compare/b3cc0c84316dd59f406a6fa23fcaf3d029910843...master

Your commit was automatically rebased without conflicts.

openjdk · 2023-10-12T19:50:30Z

@JornVernee Pushed as commit 32ac72c.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

mcimadamore and others added 13 commits July 31, 2023 10:10

8308858: FFM API and strings

cb37298

Reviewed-by: jvernee

8310048: Drop sequence layout factory with inferred count

eb90646

Reviewed-by: jvernee

8308293: A linker should expose the layouts it supports

6e2e397

Reviewed-by: jvernee

8310362: Improve composability of handle derived from layouts

d181366

Reviewed-by: mcimadamore

8310820: Remove MemorySegment::segmentOffset

201baf4

Reviewed-by: jvernee

8310893: VarHandleTestExact fails

e3d6c36

Reviewed-by: mcimadamore

8311533: SegmentAllocator::allocateArray call can be ambiguous

3aa0e02

Reviewed-by: jvernee

8312059: Clarify the documention for variadic functions

a10c182

8310646: Javadoc around prototype-less functions might be incorrect Reviewed-by: mcimadamore

8312186: TestStringEncodingFails for UTF-32

e1a8192

Reviewed-by: mcimadamore

8312615: Ensure jdk_foreign tests pass on linux-x86

854d26d

Reviewed-by: mcimadamore

8312981: Make the linker API required

d01ee34

Reviewed-by: mcimadamore

8313005: Ensure native access check can fold away

a59cdde

Reviewed-by: mcimadamore

8313265: Move the FFM API out of preview

a911f27

Reviewed-by: mcimadamore

openjdk bot added the csr Pull request needs approved CSR before integration label Aug 1, 2023

openjdk bot added nio nio-dev@openjdk.org build build-dev@openjdk.org hotspot hotspot-dev@openjdk.org core-libs core-libs-dev@openjdk.org shenandoah shenandoah-dev@openjdk.org labels Aug 1, 2023

openjdk bot removed the shenandoah shenandoah-dev@openjdk.org label Aug 1, 2023

openjdk bot added the hotspot-compiler hotspot-compiler-dev@openjdk.org label Aug 1, 2023

JornVernee added 3 commits September 27, 2023 17:13

Use abort instead of IEA when encountering wrong value for ENA attrib.

a8f0a3f

Use IAE instead of UOE for unsupported char sets

ea1b9c5

drop unneeded @compile tags from jtreg tests

2bc0a65

AlanBateman approved these changes Sep 28, 2023

View reviewed changes

JornVernee added 2 commits September 28, 2023 13:58

Merge branch 'master' into JEP22

72650c4

review @enablePreview from java/foreign/TestRestricted test

17dacbb

Remove PIP annotation from jdk.incubator.vector

cc89a51

openjdk bot removed the csr Pull request needs approved CSR before integration label Oct 6, 2023

JornVernee added 2 commits October 6, 2023 17:54

Merge branch 'master' into JEP22

e7dbce5

fix failing RestrictedMethods test

b4a7b7a

jerboaa mentioned this pull request Oct 9, 2023

8316304: (fs) Add support for BasicFileAttributes.creationTime() for Linux #15792

Closed

6 tasks

This was referenced Oct 10, 2023

8317819: Scope should reflect lifetime of underying resource (mainline) #16126

Closed

8317824: Beef up javadoc for base offset in var handles derived from layouts (mainline) #16127

Closed

update copyright years

35ca192

openjdk bot mentioned this pull request Oct 10, 2023

8317837: Leftover FFM implementation-only changes #16128

Closed

3 tasks

adjust misformatted copyright headers

5cf9e75

openjdk bot added ready Pull request is ready to be integrated and removed jep labels Oct 12, 2023

openjdk bot added the integrated Pull request has been integrated label Oct 12, 2023

openjdk bot closed this Oct 12, 2023

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Oct 12, 2023

JornVernee deleted the JEP22 branch October 12, 2023 19:53

8312522: Implementation of Foreign Function & Memory API #15103

8312522: Implementation of Foreign Function & Memory API #15103

Uh oh!

Conversation

JornVernee commented Aug 1, 2023 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issues

Reviewers

Contributors

Reviewing

Webrev

Uh oh!

bridgekeeper bot commented Aug 1, 2023

Uh oh!

openjdk bot commented Aug 1, 2023

Uh oh!

JornVernee commented Aug 1, 2023

Uh oh!

JornVernee commented Aug 1, 2023

Uh oh!

openjdk bot commented Aug 1, 2023

Uh oh!

openjdk bot commented Aug 1, 2023

Uh oh!

JornVernee commented Aug 1, 2023

Uh oh!

openjdk bot commented Aug 1, 2023

Uh oh!

JornVernee commented Aug 1, 2023

Uh oh!

ExE-Boss commented Sep 30, 2023

Uh oh!

JornVernee commented Sep 30, 2023

Uh oh!

JornVernee commented Oct 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openjdk bot commented Oct 12, 2023

Uh oh!

JornVernee commented Oct 12, 2023

Uh oh!

openjdk bot commented Oct 12, 2023

Uh oh!

openjdk bot commented Oct 12, 2023

Uh oh!

Uh oh!

JornVernee commented Aug 1, 2023 •

edited by openjdk bot

Loading

JornVernee commented Oct 6, 2023 •

edited

Loading