8262952: [macos_aarch64] os::commit_memory failure #3865

gerard-ziemski · 2021-05-04T19:34:18Z

On x86_64 macOS the following sequence works just fine:

attempt_reserve_memory_at(executable=false)
commit_memory(executable=true)

however, on aarch64 macOS it fails.

We fix the test itself to explicitly ask for executable memory when reserving it since it later commits it as executable.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8262952: [macos_aarch64] os::commit_memory failure

Reviewers

Thomas Stuefe (@tstuefe - Reviewer) ⚠️ Review applies to ad3467b
Andrew Haley (@theRealAph - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/3865/head:pull/3865
$ git checkout pull/3865

Update a local copy of the PR:
$ git checkout pull/3865
$ git pull https://git.openjdk.java.net/jdk pull/3865/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3865

View PR using the GUI difftool:
$ git pr show -t 3865

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/3865.diff

The text was updated successfully, but these errors were encountered:

bridgekeeper · 2021-05-04T19:34:58Z

👋 Welcome back gziemski! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-05-04T19:36:56Z

@gerard-ziemski The following label will be automatically applied to this pull request:

hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

…lone

mlbridge · 2021-05-07T16:20:25Z

Webrevs

theRealAph · 2021-05-10T09:44:00Z

src/hotspot/os/bsd/os_bsd.cpp

+      MACOS_ONLY(| (exec ? MAP_JIT : 0))
+      // On macOS aarch64 MAP_JIT is required if we want to commit an executable chunk
+      // later, so always reserve executable memory right from the start
+      MACOS_AARCH64_ONLY(| MAP_JIT);


Is this the right way to do it? Where do we ever map memory we want to use for code generation, but not ask for exec permission? Is that a bug?

Inside the hotspot VM code we seem to be doing the right thing, however, in the test, i.e. test/hotspot/gtest/runtime/test_os.cpp we ask for regular memory, then try to commit a chunk with elevated exec privileges, which seems to work on all other platforms, except aarch64 macOS.

The alternative, would be to fix the test in question, but since this scenario works fine on all the other platforms, I figured the proposed way to handle it in the VM would be the preferable way to fix it, so future test writers do not need to know this particular platform behavior difference and risk getting it wrong again.

I get that, but we're disabling a security feature by always mapping memory as (potentially) executable. So, we're trading runtime VM security for convenience in testing, and that strikes me as a bad bargain.

Hi @gerard-ziemski,

please fix the test instead. This would be a security issue, especially since the locations of some of our mappings are very predictable (eg CDS archive).

If this is about the "multi-release" test in test_os.cpp (377ff), just disable the exec flag there. I use alternating exec
flags to prevent the kernel from folding neighboring mappings, but actually that part is rather linux specific. Arguably we may even completely exclude the test, like I do for AIX already. Its important for Windows, somewhat less important for Linux, and covers other platforms only for completeness sake.

Wrt to MAP_JIT , what we do now is the result of long discussions: https://git.openjdk.java.net/jdk/pull/294.

Cheers, Thomas

I did struggle deciding whether it's a good compromise to make the testing more convenient and consistent among platforms by allowing memory on macOS aarch64 executable right from start, which is why I mentioned that we can (and do) remove that privilege later when we commit it, but after stepping away from this code for a moment and looking at it now, I do agree that it was bad choice.

I'll fix the test instead.

tstuefe · 2021-05-11T13:32:59Z

src/hotspot/os/bsd/os_bsd.cpp

+      MACOS_ONLY(| (exec ? MAP_JIT : 0))
+      // On macOS aarch64 MAP_JIT is required if we want to commit an executable chunk
+      // later, so always reserve executable memory right from the start
+      MACOS_AARCH64_ONLY(| MAP_JIT);


Hi @gerard-ziemski,

please fix the test instead. This would be a security issue, especially since the locations of some of our mappings are very predictable (eg CDS archive).

If this is about the "multi-release" test in test_os.cpp (377ff), just disable the exec flag there. I use alternating exec
flags to prevent the kernel from folding neighboring mappings, but actually that part is rather linux specific. Arguably we may even completely exclude the test, like I do for AIX already. Its important for Windows, somewhat less important for Linux, and covers other platforms only for completeness sake.

Wrt to MAP_JIT , what we do now is the result of long discussions: https://git.openjdk.java.net/jdk/pull/294.

Cheers, Thomas

…r, fix the test intead to ask for exececutable memory as needed

gerard-ziemski · 2021-05-11T16:02:20Z

Thank you theRealAph & Thomas for your feedback! Lesson learned.

I fixed the test instead.

tstuefe

Looks good. Thanks for fixing!

openjdk · 2021-05-11T16:36:11Z

@gerard-ziemski This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8262952: [macos_aarch64] os::commit_memory failure

Reviewed-by: stuefe, aph

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 191 new commits pushed to the master branch:

da7c846: 8264408: test_oopStorage no longer needs to disable some tests on WIN32
f6c2891: 8267229: Split runtime/Metaspace/elastic test configurations for better scalability
b60975d: 8267237: ARM32: bad AD file in matcher.cpp after 8266810
905b41a: 8265711: C1: Intrinsify Class.getModifier method
554caf3: 8251392: Consolidate Metaspace Statistics
3e97b07: 8267116: Lanai: Incorrect AlphaComposite for VolatileImage graphics
cd1c17c: 8266404: Fatal error report generated with -XX:+CrashOnOutOfMemoryError should not contain suggestion to submit a bug report
2effdd1: 8267112: JVMCI compiler modules should be kept upgradable
da4dfde: 8264777: Overload optimized FileInputStream::readAllBytes
3b11d81: 8266742: Check W^X state on possible safepoint
... and 181 more: https://git.openjdk.java.net/jdk/compare/80323b7f66541e24177d02cc668a2eb9267962b9...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

test/hotspot/gtest/runtime/test_os.cpp

…t exec memory or not

gerard-ziemski · 2021-05-12T15:11:00Z

Talking about "exec" permission of the memory in os::pd_commit_memory got me thinking about the permissions we currently set on it, which is:

int prot = exec ? PROT_READ|PROT_WRITE|PROT_EXEC : PROT_READ|PROT_WRITE

I'm sure there must be a reason, that I don't know, but why do we need it to be of "PROT_READ"? Java2Demo, for example, seems to run fine without the "read" permission on it, i.e.:

int prot = exec ? PROT_WRITE|PROT_EXEC : PROT_WRITE;

gerard-ziemski · 2021-05-17T16:21:14Z

@theRealAph Could you please sign off on this change if you are OK with my change incorporating your feedback?

gerard-ziemski · 2021-05-18T14:42:48Z

/integrate

openjdk · 2021-05-18T14:44:38Z

@gerard-ziemski Since your change was applied there have been 196 commits pushed to the master branch:

f8f40ab: 8230486: G1BarrierSetAssembler::g1_write_barrier_post unnecessarily pushes/pops new_val
9d168e2: 8266973: Migrate to ClassHierarchyIterator when enumerating subclasses
02507bc: 8267166: Remove test file vmTestbase/vm/mlvm/tools/LoadClass.java
ce88b33: 8266615: C2 incorrectly folds subtype checks involving an interface array
894547d: 8266897: com/sun/net/httpserver/FilterTest.java fails intermittently with AssertionError
da7c846: 8264408: test_oopStorage no longer needs to disable some tests on WIN32
f6c2891: 8267229: Split runtime/Metaspace/elastic test configurations for better scalability
b60975d: 8267237: ARM32: bad AD file in matcher.cpp after 8266810
905b41a: 8265711: C1: Intrinsify Class.getModifier method
554caf3: 8251392: Consolidate Metaspace Statistics
... and 186 more: https://git.openjdk.java.net/jdk/compare/80323b7f66541e24177d02cc668a2eb9267962b9...master

Your commit was automatically rebased without conflicts.

Pushed as commit fadf580.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

always use MAP_JIT when reserving memory on aarch64 macOS

e8ac515

openjdk bot added the hotspot-runtime label May 4, 2021

gerard-ziemski added 2 commits May 5, 2021

pasted the workaround in wrong place, no need for __APPLE__ check

ca0100e

move the fix to modify the const int flags, leave the exec argument a…

4a9e4a8

…lone

gerard-ziemski marked this pull request as ready for review May 7, 2021

openjdk bot added the rfr label May 7, 2021

theRealAph reviewed May 10, 2021

View changes

tstuefe suggested changes May 11, 2021

View changes

do not expose memory with exec priviledges unless explicitly asked fo…

ad3467b

…r, fix the test intead to ask for exececutable memory as needed

tstuefe approved these changes May 11, 2021

View changes

openjdk bot added the ready label May 11, 2021

theRealAph reviewed May 12, 2021

View changes

test/hotspot/gtest/runtime/test_os.cpp Outdated Show resolved Hide resolved

Loading

use the existing executable flag to indicate whether we want to commi…

63d33ef

…t exec memory or not

theRealAph approved these changes May 18, 2021

View changes

openjdk bot closed this May 18, 2021

openjdk bot added integrated and removed ready rfr labels May 18, 2021

gerard-ziemski mentioned this pull request May 18, 2021

8267339: Temporarily disable os.release_multi_mappings_vm on macOS x64 #4100

Closed

3 tasks

gerard-ziemski deleted the JDK-8262952 branch May 26, 2021

openjdk / jdk Public

8262952: [macos_aarch64] os::commit_memory failure #3865

8262952: [macos_aarch64] os::commit_memory failure #3865

gerard-ziemski commented May 4, 2021 •

edited by openjdk bot

Loading

bridgekeeper bot commented May 4, 2021

openjdk bot commented May 4, 2021

mlbridge bot commented May 7, 2021 •

edited

Loading

theRealAph May 10, 2021

gerard-ziemski May 10, 2021 •

edited

Loading

theRealAph May 11, 2021

tstuefe May 11, 2021

gerard-ziemski May 11, 2021

tstuefe May 11, 2021

gerard-ziemski commented May 11, 2021 •

edited

Loading

tstuefe left a comment

openjdk bot commented May 11, 2021 •

edited

Loading

gerard-ziemski commented May 12, 2021

gerard-ziemski commented May 17, 2021

gerard-ziemski commented May 18, 2021

openjdk bot commented May 18, 2021

openjdk / jdk Public

8262952: [macos_aarch64] os::commit_memory failure #3865

8262952: [macos_aarch64] os::commit_memory failure #3865

Conversation

gerard-ziemski commented May 4, 2021 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented May 4, 2021

openjdk bot commented May 4, 2021

mlbridge bot commented May 7, 2021 • edited Loading

Webrevs

theRealAph May 10, 2021

Choose a reason for hiding this comment

gerard-ziemski May 10, 2021 • edited Loading

Choose a reason for hiding this comment

theRealAph May 11, 2021

Choose a reason for hiding this comment

tstuefe May 11, 2021

Choose a reason for hiding this comment

gerard-ziemski May 11, 2021

Choose a reason for hiding this comment

tstuefe May 11, 2021

Choose a reason for hiding this comment

gerard-ziemski commented May 11, 2021 • edited Loading

tstuefe left a comment

openjdk bot commented May 11, 2021 • edited Loading

gerard-ziemski commented May 12, 2021

gerard-ziemski commented May 17, 2021

gerard-ziemski commented May 18, 2021

openjdk bot commented May 18, 2021

gerard-ziemski commented May 4, 2021 •

edited by openjdk bot

Loading

mlbridge bot commented May 7, 2021 •

edited

Loading

gerard-ziemski May 10, 2021 •

edited

Loading

gerard-ziemski commented May 11, 2021 •

edited

Loading

openjdk bot commented May 11, 2021 •

edited

Loading