8253795: Implementation of JEP 391: macOS/AArch64 Port #715

VladimirKempik · 2021-12-13T20:11:31Z

Initial version of JEP-391 backport to jdk11u.
Build system changes are mostly clean, except cds disabling part, it's copy&paste from aix part.
Things needing attention: os_bsd_aarch64.cpp and W^X transitions.
serviceability agent is mostly clean.
This passed GHA_tier1 testing.
Full regression testing is running now on Azul's infra, will report/update PR when done.
Update: TCK passed, full regression testing is fine on intel platforms, macos/aarch64/openjdk11 is good compared to macos/aarch64/zulu11
Sharing this PR slightly earlier to let other interested parties to run the tests too.
Example of cross-building on intel mac (needs Xcode12/13):
sh configure --with-boot-jdk=/Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home/ --with-build-jdk=/Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home/ --disable-warnings-as-errors --openjdk-target=aarch64-apple-darwin

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issues

JDK-8253795: Implementation of JEP 391: macOS/AArch64 Port
JDK-8253816: Support macOS W^X
JDK-8253817: Support macOS Aarch64 ABI in Interpreter
JDK-8253818: Support macOS Aarch64 ABI for compiled wrappers
JDK-8253819: Implement os/cpu for macOS/AArch64
JDK-8253839: Update tests and JDK code for macOS/Aarch64
JDK-8254941: Implement Serviceability Agent for macOS/AArch64
JDK-8255776: Change build system for macOS/AArch64
JDK-8262903: [macos_aarch64] Thread::current() called on detached thread
JDK-8262896: [macos_aarch64] Crash in jni_fast_GetLongField

Reviewers

Andrew Dinn (@adinn - Reviewer) ⚠️ Review applies to 351be96
Bernhard Urban-Forster (@lewurm - no project role) ⚠️ Review applies to 351be96
Andrew Haley (@theRealAph - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk11u-dev pull/715/head:pull/715
$ git checkout pull/715

Update a local copy of the PR:
$ git checkout pull/715
$ git pull https://git.openjdk.java.net/jdk11u-dev pull/715/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 715

View PR using the GUI difftool:
$ git pr show -t 715

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk11u-dev/pull/715.diff

bridgekeeper · 2021-12-13T20:12:58Z

👋 Welcome back vkempik! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-12-13T20:14:11Z

This backport pull request has now been updated with issues from the original commit.

openjdk · 2021-12-13T20:17:14Z

@VladimirKempik
Updating description of additional solved issue: 8253816: Support macOS W^X.

openjdk · 2021-12-13T20:18:38Z

@VladimirKempik
Updating description of additional solved issue: 8253817: Support macOS Aarch64 ABI in Interpreter.

openjdk · 2021-12-13T20:20:00Z

@VladimirKempik
Updating description of additional solved issue: 8253818: Support macOS Aarch64 ABI for compiled wrappers.

openjdk · 2021-12-13T20:21:26Z

@VladimirKempik
Updating description of additional solved issue: 8253819: Implement os/cpu for macOS/AArch64.

openjdk · 2021-12-13T20:22:50Z

@VladimirKempik
Updating description of additional solved issue: 8253839: Update tests and JDK code for macOS/Aarch64.

openjdk · 2021-12-13T20:24:33Z

@VladimirKempik
Adding additional issue to issue list: 8253841: jextract --source mode is broken.

openjdk · 2021-12-13T20:26:04Z

@VladimirKempik
Updating description of additional solved issue: 8255776: Change build system for macOS/AArch64.

openjdk · 2021-12-13T20:27:24Z

@VladimirKempik
Updating description of additional solved issue: 8262903: [macos_aarch64] Thread::current() called on detached thread.

openjdk · 2021-12-13T20:28:54Z

@VladimirKempik
Updating description of additional solved issue: 8254941: Implement Serviceability Agent for macOS/AArch64.

openjdk · 2021-12-13T20:30:16Z

@VladimirKempik
Removing additional issue from issue list: 8253841.

mlbridge · 2021-12-13T20:36:21Z

Webrevs

VladimirKempik · 2021-12-14T06:49:23Z

Testing revealed the need to include https://bugs.openjdk.java.net/browse/JDK-8262896 into this PR as well.

8262896: [macos_aarch64] Crash in jni_fast_GetLongField

VladimirKempik · 2021-12-14T06:52:11Z

/issue 8262896

openjdk · 2021-12-14T06:53:34Z

⚠️ @VladimirKempik This pull request contains merges that bring in commits not present in the target repository. Since this is not a "merge style" pull request, these changes will be squashed when this pull request in integrated. If this is your intention, then please ignore this message. If you want to preserve the commit structure, you must change the title of this pull request to Merge <project>:<branch> where <project> is the name of another project in the OpenJDK organization (for example Merge jdk:master).

openjdk · 2021-12-14T06:53:44Z

@VladimirKempik
Adding additional issue to issue list: 8262896: [macos_aarch64] Crash in jni_fast_GetLongField.

VladimirKempik · 2021-12-14T08:28:03Z

Together with #716 the hotspot can be built without warnings, some warning in splashscreen yet.

theRealAph · 2021-12-14T10:41:13Z

This is an extremely difficult patch to review. I'll be soliciting other people's advice to help understand the risk versus reward ratio to bring this in to mainline.
One thing to consider is that, given the good performance of Rosetta, x86 binaries can be used. So, in effect this is only a performance patch for Mac/AArch64. On the other hand, Mac/AArch64 has been working well for some time on the AArch64-port/11u repo.

VladimirKempik · 2022-02-05T13:05:40Z

Crashlog with hsdis - https://pastebin.com/5kz2vfEH

SIGBUS (0xa) at pc=0x000000012100bbfc, pid=76024, tid=39683
siginfo: si_signo: 10 (SIGBUS), si_code: 1 (BUS_ADRALN), si_addr: 0x000000012100bbfc

#128
0x000000012100bbc0: movk x8, #0x2100, lsl #16
0x000000012100bbc4: movk x8, #0x1, lsl #32
0x000000012100bbc8: br x8
0x000000012100bbcc: ldp x22, x24, [x29, #-72]
0x000000012100bbd0: ldur x12, [x29, #-24]
0x000000012100bbd4: ldr x10, [x12, #80]
0x000000012100bbd8: add x0, x28, #0x350
0x000000012100bbdc: adr x8, 0x000000012100bbfc
0x000000012100bbe0: str x8, [x28, #824]
0x000000012100bbe4: str x20, [x28, #816]
0x000000012100bbe8: str x29, [x28, #832]
0x000000012100bbec: orr x8, xzr, #0x4
0x000000012100bbf0: add x9, x28, #0x3c8
0x000000012100bbf4: stlr w8, [x9]
0x000000012100bbf8: blr x10
0x000000012100bbfc: isb
0x000000012100bc00: ldur x12, [x29, #-24]
0x000000012100bc04: sub x8, x20, #0x20
0x000000012100bc08: and sp, x8, #0xfffffffffffffff0
0x000000012100bc0c: str d0, [x20, #-16]!
0x000000012100bc10: str xzr, [x20, #-8]!
0x000000012100bc14: str x0, [x20, #-8]!
0x000000012100bc18: mov x8, #0x5

Pretty strange, @theRealAph, any idea ?

a snippet from macos's crashlog for java
Crashed Thread: 21 Java: MainThread

Exception Type: EXC_BAD_ACCESS (SIGABRT)
Exception Codes: KERN_PROTECTION_FAILURE at 0x000000012103bdac
Exception Note: EXC_CORPSE_NOTIFY

VM Regions Near 0x12103bdac:
MALLOC_SMALL 120800000-121000000 [ 8192K] rw-/rwx SM=PRV
--> VM_ALLOCATE 121000000-121270000 [ 2496K] rwx/rwx SM=ZER
VM_ALLOCATE 121270000-121594000 [ 3216K] ---/rw

38 libjvm.dylib 0x0000000107541f1c JVM_handle_bsd_signal + 820 (os_bsd_aarch64.cpp:471)
39 libjvm.dylib 0x000000010753f9d8 signalHandler(int, __siginfo*, void*) + 56 (os_bsd.cpp:2879)
40 libsystem_platform.dylib 0x0000000194868c44 _sigtramp + 56
41 ??? 0x000000012100bbfc 0 + 4848663548
42 ??? 0x0000000121006ab0 0 + 4848642736
43 ??? 0x000000012100719c 0 + 4848644508
44 ??? 0x0000000121006ee0 0 + 4848643808
45 ??? 0x0000000121006ee0 0 + 4848643808
46 ??? 0x0000000121006ee0 0 + 4848643808
47 ??? 0x000000012100719c 0 + 4848644508
48 ??? 0x0000000121000144 0 + 4848615748
49 libjvm.dylib 0x0000000107322c9c JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*) + 548 (javaCalls.cpp:444)

VladimirKempik · 2022-02-05T15:30:10Z

Looks like it was two missing WX transitions in JNI_QUICK_ENTRY and JVM_QUICK_ENTRY
These transitions are present in zulu11
JNI_QUICK_ENTRY and JVM_QUICK_ENTRY macros are missing in jdk17/upstream.

That explains everything, except the macos version reproducibility, which is probably due to the difference in library calls leaving different w^x state up on exit.

@ahmedmuhsin, when you will have a time, please check the updated PR on macos 11.0.1 or 11.2.1, for me it doesn't crash anymore

ahmedmuhsin · 2022-02-07T22:52:32Z

@VladimirKempik I tested again on all the OS versions that I mentioned above and everything looks very good. No crashes. Thanks!

VladimirKempik · 2022-02-08T06:48:00Z

@rnkovacs , parts of your jep-388 backport ( VladimirKempik@7bdb8ac7f9f9#diff-0f4150a9c607ccd590bf256daa800c0276144682a92bc6bdced5e8bc1bb81f3aL1495, VladimirKempik@7bdb8ac7f9f9#diff-0f4150a9c607ccd590bf256daa800c0276144682a92bc6bdced5e8bc1bb81f3aL1558, VladimirKempik@7bdb8ac7f9f9#diff-0f4150a9c607ccd590bf256daa800c0276144682a92bc6bdced5e8bc1bb81f3aL1671 ) results in more warning for this PR:

/Users/runner/work/jdk11u-dev/jdk11u-dev/jdk/src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp:1495:52: error: format specifies type 'unsigned long long' but the argument has type 'uintptr_t' (aka 'unsigned long') [-Werror,-Wformat]
snprintf(buffer, sizeof(buffer), PTR64_FORMAT, imm64);
~~~~~~~~~~~~ ^~~~~
1 error generated.

was these three changes really needed or can I revert them ?

reverting these three lines will allow me to fix macos-aarch64 github CI job ( https://github.com/VladimirKempik/jdk11u-dev/runs/5106043614?check_suite_focus=true )

theRealAph · 2022-02-08T10:00:50Z

On 2/8/22 06:48, VladimirKempik wrote: @rnkovacs , parts of your jep-388 backport reverted ( VladimirKempik@7bdb8ac7f9f9#diff-0f4150a9c607ccd590bf256daa800c0276144682a92bc6bdced5e8bc1bb81f3aL1495, VladimirKempik@7bdb8ac7f9f9#diff-0f4150a9c607ccd590bf256daa800c0276144682a92bc6bdced5e8bc1bb81f3aL1558, VladimirKempik@7bdb8ac7f9f9#diff-0f4150a9c607ccd590bf256daa800c0276144682a92bc6bdced5e8bc1bb81f3aL1671 ) results in more warning for this PR: /Users/runner/work/jdk11u-dev/jdk11u-dev/jdk/src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp:1495:52: error: format specifies type 'unsigned long long' but the argument has type 'uintptr_t' (aka 'unsigned long') [-Werror,-Wformat] snprintf(buffer, sizeof(buffer), PTR64_FORMAT, imm64); ~~~~~~~~~~~~ ^~~~~ 1 error generated. was these three changes really needed or can I revert them ?

What does this refer to? I can't find any such thing in the current state of the patch.

VladimirKempik · 2022-02-08T10:25:57Z

These three line changes are part of jep-388 (win-arm64) backport to jdk11u-dev, they cause troubles ( warnings which are under Werror becomes an issue)
these changes aren't present in upstream (jdk19), for example - https://github.com/openjdk/jdk/blob/master/src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp#L1316

so I wonder why did they became part of jep-388 backport to jdk11u-dev
if there is no good reason, then I would like to revert these changes as part of this PR.

theRealAph · 2022-02-08T10:33:10Z

so I wonder why did they became part of jep-388 backport to jdk11u-dev if there is no good reason, then I would like to revert these changes as part of this PR.

There can be no good reason for an incorrect change, so please do that.

VladimirKempik · 2022-02-08T10:33:52Z

Ok.
Also. After latest merge with master this can now be crosscompiled without -with-extra-"c/ld/cxx"flags

ahmedmuhsin · 2022-02-08T21:10:34Z

@VladimirKempik should I submit a PR to backport 8241004: NMT tests fail on unaligned thread size with debug build or is it already on your radar?

The backport solves issue 2 from my post here: #715 (comment)

VladimirKempik · 2022-02-08T21:12:37Z

Please proceed with 8241004, I have few other macarm related issues on my radar atm.

VladimirKempik · 2022-02-09T07:58:24Z

@theRealAph what do you think, is this PR ready to go jdk11u-dev ?
it would be good to merge it some time before rdp2 to have a time for non-critical fixes.

theRealAph

This looks good enough.

VladimirKempik · 2022-02-14T18:39:16Z

Oracle has pulled this into their java11 - https://bugs.openjdk.java.net/browse/JDK-8275573

@GoeLin is there anything holding us from getting approval for 11u ?

VladimirKempik · 2022-02-15T14:07:21Z

/integrate

openjdk · 2022-02-15T14:10:09Z

Going to push as commit eb0708f.
Since your change was applied there have been 13 commits pushed to the master branch:

fe17b44: 8280414: Memory leak in DefaultProxySelector
fd7d0d3: 8280526: x86_32 Math.sqrt performance regression with -XX:UseSSE={0,1}
bd420d2: 8279076: C2: Bad AD file when matching SqrtF with UseSSE=0
29eb422: 8281520: JFR: A wrong parameter is passed to the constructor of LeakKlassWriter
d79e710: 8281599: test/lib/jdk/test/lib/KnownOIDs.java is redundant since JDK-8268801
6c26ba2: 8190748: java/text/Format/DateFormat/DateFormatTest.java and NonGregorianFormatTest fail intermittently
f4e14a6: 8281061: [s390] JFR runs into assertions while validating interpreter frames
668c200: 8280155: [PPC64, s390] frame size checks are not yet correct
6921496: 8279924: [PPC64, s390] implement frame::is_interpreted_frame_valid checks
1db4ed1: 8261205: AssertionError: Cannot add metadata to an intersection type
... and 3 more: https://git.openjdk.java.net/jdk11u-dev/compare/afb94aeae8ffab20ca07dd0fad3765fd65a15255...master

Your commit was automatically rebased without conflicts.

openjdk · 2022-02-15T14:10:32Z

@VladimirKempik Pushed as commit eb0708f.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Vladimir Kempik added 3 commits December 13, 2021 16:33

Initial commit

03aaf5b

Fix up linux_aarch64

939c919

Trailing white space removal

acb947f

openjdk bot changed the title ~~Backport dbc9e4b50cdda35f5712deaf440c49f50b9edc96~~ 8253795: Implementation of JEP 391: macOS/AArch64 Port Dec 13, 2021

openjdk bot added backport rfr Pull request is ready for review labels Dec 13, 2021

8262896: [macos_aarch64] Crash in jni_fast_GetLongField

6270632

Merge pull request #1 from VladimirKempik/JEP391_and_8262896

5bc3427

8262896: [macos_aarch64] Crash in jni_fast_GetLongField

Vladimir Kempik added 3 commits December 14, 2021 10:00

Exclude test CompressedClassPointers.java on macarm

fc77811

Fix warning about c++11 extension

5329585

remove unused at_rsp from aarch64

34cb67d

Add two missing WX transitions

787ccf8

Merge branch 'master' into JDK-8253795

fe73bf5

Fix wrong patch from jep-388 backport

6d03795

This was referenced Feb 8, 2022

Backport 0b20eafb30cc1de372141f2a427d13fdc6565e3a openjdk-bots/jdk11u-dev#1

Closed

8241004: NMT tests fail on unaligned thread size with debug build #812

Closed

mseele mentioned this pull request Feb 9, 2022

Provide JDK/JRE 8 and 11 builds for Apple Silicon (M1/ARM/aarch64) adoptium/adoptium#96

Open

theRealAph approved these changes Feb 9, 2022

View reviewed changes

This was referenced Feb 9, 2022

8262894: [macos_aarch64] SIGBUS in Assembler::ld_st2 #816

Closed

8266889: [macosx-aarch64] Crash with SIGBUS in MarkActivationClosure::do_code_blob during vmTestbase/nsk/jvmti/.../bi04t002 test run #817

Closed

openjdk bot added the integrated Pull request has been integrated label Feb 15, 2022

openjdk bot closed this Feb 15, 2022

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Feb 15, 2022

ahmedmuhsin mentioned this pull request Feb 26, 2022

8282372: [11] build issue on MacOS/aarch64 12.2.1 using Xcode 13.1: call to 'log2_intptr' is ambiguous #841

Closed

3 tasks

VladimirKempik mentioned this pull request Feb 12, 2023

8276799: Implementation of JEP 422: Linux/RISC-V Port openjdk/riscv-port-jdk17u#1

Merged

3 tasks

8253795: Implementation of JEP 391: macOS/AArch64 Port #715

8253795: Implementation of JEP 391: macOS/AArch64 Port #715

VladimirKempik commented Dec 13, 2021 •

edited by openjdk bot

bridgekeeper bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

mlbridge bot commented Dec 13, 2021 •

edited

VladimirKempik commented Dec 14, 2021 •

edited

VladimirKempik commented Dec 14, 2021

openjdk bot commented Dec 14, 2021

openjdk bot commented Dec 14, 2021

VladimirKempik commented Dec 14, 2021

theRealAph commented Dec 14, 2021

VladimirKempik commented Feb 5, 2022 •

edited

VladimirKempik commented Feb 5, 2022 •

edited

ahmedmuhsin commented Feb 7, 2022

VladimirKempik commented Feb 8, 2022 •

edited

theRealAph commented Feb 8, 2022 via email

VladimirKempik commented Feb 8, 2022 •

edited

theRealAph commented Feb 8, 2022

VladimirKempik commented Feb 8, 2022 •

edited

ahmedmuhsin commented Feb 8, 2022

VladimirKempik commented Feb 8, 2022

VladimirKempik commented Feb 9, 2022

theRealAph left a comment

VladimirKempik commented Feb 14, 2022

VladimirKempik commented Feb 15, 2022

openjdk bot commented Feb 15, 2022

openjdk bot commented Feb 15, 2022

8253795: Implementation of JEP 391: macOS/AArch64 Port #715

8253795: Implementation of JEP 391: macOS/AArch64 Port #715

Conversation

VladimirKempik commented Dec 13, 2021 • edited by openjdk bot

Progress

Issues

Reviewers

Reviewing

bridgekeeper bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

openjdk bot commented Dec 13, 2021

mlbridge bot commented Dec 13, 2021 • edited

Webrevs

VladimirKempik commented Dec 14, 2021 • edited

VladimirKempik commented Dec 14, 2021

openjdk bot commented Dec 14, 2021

openjdk bot commented Dec 14, 2021

VladimirKempik commented Dec 14, 2021

theRealAph commented Dec 14, 2021

VladimirKempik commented Feb 5, 2022 • edited

VladimirKempik commented Feb 5, 2022 • edited

ahmedmuhsin commented Feb 7, 2022

VladimirKempik commented Feb 8, 2022 • edited

theRealAph commented Feb 8, 2022 via email

VladimirKempik commented Feb 8, 2022 • edited

theRealAph commented Feb 8, 2022

VladimirKempik commented Feb 8, 2022 • edited

ahmedmuhsin commented Feb 8, 2022

VladimirKempik commented Feb 8, 2022

VladimirKempik commented Feb 9, 2022

theRealAph left a comment

Choose a reason for hiding this comment

VladimirKempik commented Feb 14, 2022

VladimirKempik commented Feb 15, 2022

openjdk bot commented Feb 15, 2022

openjdk bot commented Feb 15, 2022

VladimirKempik commented Dec 13, 2021 •

edited by openjdk bot

mlbridge bot commented Dec 13, 2021 •

edited

VladimirKempik commented Dec 14, 2021 •

edited

VladimirKempik commented Feb 5, 2022 •

edited

VladimirKempik commented Feb 5, 2022 •

edited

VladimirKempik commented Feb 8, 2022 •

edited

VladimirKempik commented Feb 8, 2022 •

edited

VladimirKempik commented Feb 8, 2022 •

edited