Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8262952: [macos_aarch64] os::commit_memory failure #3865

Closed
wants to merge 5 commits into from

Conversation

@gerard-ziemski
Copy link

@gerard-ziemski gerard-ziemski commented May 4, 2021

On x86_64 macOS the following sequence works just fine:

attempt_reserve_memory_at(executable=false)
commit_memory(executable=true)

however, on aarch64 macOS it fails.

We fix the test itself to explicitly ask for executable memory when reserving it since it later commits it as executable.


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed

Issue

  • JDK-8262952: [macos_aarch64] os::commit_memory failure

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/3865/head:pull/3865
$ git checkout pull/3865

Update a local copy of the PR:
$ git checkout pull/3865
$ git pull https://git.openjdk.java.net/jdk pull/3865/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3865

View PR using the GUI difftool:
$ git pr show -t 3865

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/3865.diff

@bridgekeeper
Copy link

@bridgekeeper bridgekeeper bot commented May 4, 2021

👋 Welcome back gziemski! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

Loading

@openjdk
Copy link

@openjdk openjdk bot commented May 4, 2021

@gerard-ziemski The following label will be automatically applied to this pull request:

  • hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

Loading

@gerard-ziemski gerard-ziemski marked this pull request as ready for review May 7, 2021
@openjdk openjdk bot added the rfr label May 7, 2021
@mlbridge
Copy link

@mlbridge mlbridge bot commented May 7, 2021

Webrevs

Loading

MACOS_ONLY(| (exec ? MAP_JIT : 0))
// On macOS aarch64 MAP_JIT is required if we want to commit an executable chunk
// later, so always reserve executable memory right from the start
MACOS_AARCH64_ONLY(| MAP_JIT);
Copy link
Contributor

@theRealAph theRealAph May 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the right way to do it? Where do we ever map memory we want to use for code generation, but not ask for exec permission? Is that a bug?

Loading

Copy link
Author

@gerard-ziemski gerard-ziemski May 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inside the hotspot VM code we seem to be doing the right thing, however, in the test, i.e. test/hotspot/gtest/runtime/test_os.cpp we ask for regular memory, then try to commit a chunk with elevated exec privileges, which seems to work on all other platforms, except aarch64 macOS.

The alternative, would be to fix the test in question, but since this scenario works fine on all the other platforms, I figured the proposed way to handle it in the VM would be the preferable way to fix it, so future test writers do not need to know this particular platform behavior difference and risk getting it wrong again.

Loading

Copy link
Contributor

@theRealAph theRealAph May 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get that, but we're disabling a security feature by always mapping memory as (potentially) executable. So, we're trading runtime VM security for convenience in testing, and that strikes me as a bad bargain.

Loading

Copy link
Member

@tstuefe tstuefe May 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @gerard-ziemski,

please fix the test instead. This would be a security issue, especially since the locations of some of our mappings are very predictable (eg CDS archive).

If this is about the "multi-release" test in test_os.cpp (377ff), just disable the exec flag there. I use alternating exec
flags to prevent the kernel from folding neighboring mappings, but actually that part is rather linux specific. Arguably we may even completely exclude the test, like I do for AIX already. Its important for Windows, somewhat less important for Linux, and covers other platforms only for completeness sake.

Wrt to MAP_JIT , what we do now is the result of long discussions: https://git.openjdk.java.net/jdk/pull/294.

Cheers, Thomas

Loading

Copy link
Author

@gerard-ziemski gerard-ziemski May 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did struggle deciding whether it's a good compromise to make the testing more convenient and consistent among platforms by allowing memory on macOS aarch64 executable right from start, which is why I mentioned that we can (and do) remove that privilege later when we commit it, but after stepping away from this code for a moment and looking at it now, I do agree that it was bad choice.

I'll fix the test instead.

Loading

MACOS_ONLY(| (exec ? MAP_JIT : 0))
// On macOS aarch64 MAP_JIT is required if we want to commit an executable chunk
// later, so always reserve executable memory right from the start
MACOS_AARCH64_ONLY(| MAP_JIT);
Copy link
Member

@tstuefe tstuefe May 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @gerard-ziemski,

please fix the test instead. This would be a security issue, especially since the locations of some of our mappings are very predictable (eg CDS archive).

If this is about the "multi-release" test in test_os.cpp (377ff), just disable the exec flag there. I use alternating exec
flags to prevent the kernel from folding neighboring mappings, but actually that part is rather linux specific. Arguably we may even completely exclude the test, like I do for AIX already. Its important for Windows, somewhat less important for Linux, and covers other platforms only for completeness sake.

Wrt to MAP_JIT , what we do now is the result of long discussions: https://git.openjdk.java.net/jdk/pull/294.

Cheers, Thomas

Loading

…r, fix the test intead to ask for exececutable memory as needed
@gerard-ziemski
Copy link
Author

@gerard-ziemski gerard-ziemski commented May 11, 2021

Thank you theRealAph & Thomas for your feedback! Lesson learned.

I fixed the test instead.

Loading

Copy link
Member

@tstuefe tstuefe left a comment

Looks good. Thanks for fixing!

Loading

@openjdk
Copy link

@openjdk openjdk bot commented May 11, 2021

@gerard-ziemski This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8262952: [macos_aarch64] os::commit_memory failure

Reviewed-by: stuefe, aph

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 191 new commits pushed to the master branch:

  • da7c846: 8264408: test_oopStorage no longer needs to disable some tests on WIN32
  • f6c2891: 8267229: Split runtime/Metaspace/elastic test configurations for better scalability
  • b60975d: 8267237: ARM32: bad AD file in matcher.cpp after 8266810
  • 905b41a: 8265711: C1: Intrinsify Class.getModifier method
  • 554caf3: 8251392: Consolidate Metaspace Statistics
  • 3e97b07: 8267116: Lanai: Incorrect AlphaComposite for VolatileImage graphics
  • cd1c17c: 8266404: Fatal error report generated with -XX:+CrashOnOutOfMemoryError should not contain suggestion to submit a bug report
  • 2effdd1: 8267112: JVMCI compiler modules should be kept upgradable
  • da4dfde: 8264777: Overload optimized FileInputStream::readAllBytes
  • 3b11d81: 8266742: Check W^X state on possible safepoint
  • ... and 181 more: https://git.openjdk.java.net/jdk/compare/80323b7f66541e24177d02cc668a2eb9267962b9...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

Loading

@openjdk openjdk bot added the ready label May 11, 2021
test/hotspot/gtest/runtime/test_os.cpp Outdated Show resolved Hide resolved
Loading
@gerard-ziemski
Copy link
Author

@gerard-ziemski gerard-ziemski commented May 12, 2021

Talking about "exec" permission of the memory in os::pd_commit_memory got me thinking about the permissions we currently set on it, which is:

int prot = exec ? PROT_READ|PROT_WRITE|PROT_EXEC : PROT_READ|PROT_WRITE

I'm sure there must be a reason, that I don't know, but why do we need it to be of "PROT_READ"? Java2Demo, for example, seems to run fine without the "read" permission on it, i.e.:

int prot = exec ? PROT_WRITE|PROT_EXEC : PROT_WRITE;

Loading

@gerard-ziemski
Copy link
Author

@gerard-ziemski gerard-ziemski commented May 17, 2021

@theRealAph Could you please sign off on this change if you are OK with my change incorporating your feedback?

Loading

@gerard-ziemski
Copy link
Author

@gerard-ziemski gerard-ziemski commented May 18, 2021

/integrate

Loading

@openjdk openjdk bot closed this May 18, 2021
@openjdk openjdk bot added integrated and removed ready rfr labels May 18, 2021
@openjdk
Copy link

@openjdk openjdk bot commented May 18, 2021

@gerard-ziemski Since your change was applied there have been 196 commits pushed to the master branch:

  • f8f40ab: 8230486: G1BarrierSetAssembler::g1_write_barrier_post unnecessarily pushes/pops new_val
  • 9d168e2: 8266973: Migrate to ClassHierarchyIterator when enumerating subclasses
  • 02507bc: 8267166: Remove test file vmTestbase/vm/mlvm/tools/LoadClass.java
  • ce88b33: 8266615: C2 incorrectly folds subtype checks involving an interface array
  • 894547d: 8266897: com/sun/net/httpserver/FilterTest.java fails intermittently with AssertionError
  • da7c846: 8264408: test_oopStorage no longer needs to disable some tests on WIN32
  • f6c2891: 8267229: Split runtime/Metaspace/elastic test configurations for better scalability
  • b60975d: 8267237: ARM32: bad AD file in matcher.cpp after 8266810
  • 905b41a: 8265711: C1: Intrinsify Class.getModifier method
  • 554caf3: 8251392: Consolidate Metaspace Statistics
  • ... and 186 more: https://git.openjdk.java.net/jdk/compare/80323b7f66541e24177d02cc668a2eb9267962b9...master

Your commit was automatically rebased without conflicts.

Pushed as commit fadf580.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants