8265292: [macos_aarch64] java/foreign/TestDowncall.java crashes with SIGBUS #3921

AntonKozlov · 2021-05-07T15:38:24Z

Please review a fix for the intermittent crash. It is caused by a mistake in the ProgrammableInvoker::invoke_native, the wrong order of W^X and JavaThread state transition. We need WXExec since we are about to call a generated stub. But we need to switch to WXExec only after JavaThread state change. The thread state change may trigger a safepoint, that would need to do bookkeeping in the codecache (MarkActivationClosure::do_code_blob from the bug). So the fix is to change JavaThread state first, then change WX.

The fix was verified with the help of https://bugs.openjdk.java.net/browse/JDK-8266742. The new check catches all test failures reported by 8265292, 8265183, 8265182. I've verified tests pass after the fix with that new check enabled.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issues

JDK-8265292: [macos_aarch64] java/foreign/TestDowncall.java crashes with SIGBUS
JDK-8265183: [macos_aarch64] java/foreign/TestIntrinsics.java crashes with SIGBUS
JDK-8265182: [macos_aarch64] java/foreign/TestUpcall.java crashes with SIGBUS

Reviewers

David Holmes (@dholmes-ora - Reviewer) ⚠️ Review applies to cb9723b

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/3921/head:pull/3921
$ git checkout pull/3921

Update a local copy of the PR:
$ git checkout pull/3921
$ git pull https://git.openjdk.java.net/jdk pull/3921/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3921

View PR using the GUI difftool:
$ git pr show -t 3921

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/3921.diff

bridgekeeper · 2021-05-07T15:39:25Z

👋 Welcome back akozlov! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

AntonKozlov · 2021-05-07T15:43:02Z

/solves 8265183
/solves 8265182

openjdk · 2021-05-07T15:43:19Z

@AntonKozlov The following label will be automatically applied to this pull request:

hotspot

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

openjdk · 2021-05-07T15:43:33Z

@AntonKozlov
Adding additional issue to solves list: 8265183: [macos_aarch64] java/foreign/TestIntrinsics.java crashes with SIGBUS.

openjdk · 2021-05-07T15:44:39Z

@AntonKozlov
Adding additional issue to solves list: 8265182: [macos_aarch64] java/foreign/TestUpcall.java crashes with SIGBUS.

mlbridge · 2021-05-07T15:49:16Z

Webrevs

01: Full - Incremental (d46cd678)
00: Full (cb9723b7)

theRealAph · 2021-05-08T09:16:11Z

src/hotspot/share/prims/universalNativeInvoker.cpp

  ThreadToNativeFromVM ttnfvm(thread);
+  MACOS_AARCH64_ONLY(ThreadWXEnable wx(WXExec, thread));


I think we need a comment here, not just in the bug report. if this confused you, it'll surely be enough to confuse a maintainer. Some think like this, perhaps?

"We need WXExec because we are about to call a generated stub. But we need to switch to WXExec only after JavaThread state change. The thread state change may trigger a safepoint, that would need to do bookkeeping in the codecache. See JDK-8265292."

I'm also wondering if it would be better to enable writes in the methods that actually write to the nmethod, WDYT?

And incidentally, this seems to be rather error prone. Wouldn't it be simpler to check W^X status in all of the nmethod accessors, and change it when actually required, rather than hoping that it's already in the correct state?

Thanks, I've added the comment here, and also one near VM entries. I would like to avoid enabling write near writing itself. There are too many such places, and either we'll cover a rather large part of hotspot with them (with trial and fix like here, also having overhead from going back and forth), or we'll start clustering them in bigger write-contexts. The current approach is to have the largest write-context possible: the whole JVM should be able to write. Therefore, the bug is not that we didn't get to WXWrite when we are trying to write, the bug is that we were not in the right state.

Also to @dholmes-ora. Sorry for failing to provide a clear principle in the first place. I really assumed that we were on the same page until reviews of JDK-8262896. Since then I'm preparing a kind of doc, although it's going hard. Thank you for still asking questions :) I'm going to include discussed topics into that doc.

dholmes-ora

If this fixes the problem - great, approved.

But this just highlights the limited understanding around W^X and thread states IMO.

Thanks,
David

openjdk · 2021-05-10T05:18:32Z

@AntonKozlov This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8265292: [macos_aarch64] java/foreign/TestDowncall.java crashes with SIGBUS
8265183: [macos_aarch64] java/foreign/TestIntrinsics.java crashes with SIGBUS
8265182: [macos_aarch64] java/foreign/TestUpcall.java crashes with SIGBUS

Reviewed-by: dholmes

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 61 new commits pushed to the master branch:

2d2cd78: 8266761: AssertionError in sun.net.httpserver.ServerImpl.responseCompleted
9c9c47e: 8266813: Shenandoah: Use shorter instruction sequence for checking if marking in progress
0344e75: 8266794: Remove dead code notify_allocation_jvmti_allocation_event
9e6e222: 8266892: avoid maybe-uninitialized gcc warnings on linux s390x
6575566: 8266787: Potential overflow of pointer arithmetic in G1ArchiveAllocator
8468001: 8263452: Javac slow compilation due to algorithmic complexity
67cb22a: 8266601: Fix bugs in AddLNode::Ideal transformations
18e9d28: 8266676: G1: Remove dead code init_node_id_to_index_map()
0e7bdae: 8265062: Remove duplication constant MaxTextureSize
10a049e: 8265956: JVM crashes when matching LShiftVB Node
... and 51 more: https://git.openjdk.java.net/jdk/compare/c665dba591ae5c15c9ca49e14d1aaa4eea38e7ae...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

mlbridge · 2021-05-14T18:31:00Z

Mailing list message from Andrew Haley on hotspot-dev:

On 5/10/21 6:19 AM, David Holmes wrote:

On Fri, 7 May 2021 15:38:24 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

Please review a fix for the intermittent crash. It is caused by a mistake in the ProgrammableInvoker::invoke_native, the wrong order of W^X and JavaThread state transition. We need WXExec since we are about to call a generated stub. But we need to switch to WXExec only after JavaThread state change. The thread state change may trigger a safepoint, that would need to do bookkeeping in the codecache (MarkActivationClosure::do_code_blob from the bug). So the fix is to change JavaThread state first, then change WX.

The fix was verified with the help of https://bugs.openjdk.java.net/browse/JDK-8266742. The new check catches all test failures reported by 8265292, 8265183, 8265182. I've verified tests pass after the fix with that new check enabled.

If this fixes the problem - great, approved.

But this just highlights the limited understanding around W^X and thread states IMO.

Yes, that's what I was getting at. This looks to me like an instance of
the Temporal Coupling design smell.

AntonKozlov · 2021-05-17T10:43:35Z

This looks to me like an instance of the Temporal Coupling design smell.

I would agree if we'd change the W^X as necessary before an appropriate action. But we don't, we maintain a property "JVM's runtime (C++) code runs in WXWrite" (now it is specified in one of the comments). And the change restores this property which was just broken inadvertently.

theRealAph · 2021-05-18T08:49:22Z

This looks to me like an instance of the Temporal Coupling design smell.

I would agree if we'd change the W^X as necessary before an appropriate action. But we don't, we maintain a property "JVM's runtime (C++) code runs in WXWrite" (now it is specified in one of the comments). And the change restores this property which was just broken inadvertently.

Sure, but this code has not been in mainline for very long. I have no objection to this patch being committed, which is obviously necessary, but I agree with David Holmes. If we're quick we can get this fixed before it ever reaches a release.

AntonKozlov · 2021-05-18T18:44:56Z

OK, thanks! I still don't think there is something needs fixing, but a consensus is necessary, I agree.

/integrate

openjdk · 2021-05-18T18:55:04Z

@AntonKozlov Since your change was applied there have been 159 commits pushed to the master branch:

fadf580: 8262952: [macos_aarch64] os::commit_memory failure
f8f40ab: 8230486: G1BarrierSetAssembler::g1_write_barrier_post unnecessarily pushes/pops new_val
9d168e2: 8266973: Migrate to ClassHierarchyIterator when enumerating subclasses
02507bc: 8267166: Remove test file vmTestbase/vm/mlvm/tools/LoadClass.java
ce88b33: 8266615: C2 incorrectly folds subtype checks involving an interface array
894547d: 8266897: com/sun/net/httpserver/FilterTest.java fails intermittently with AssertionError
da7c846: 8264408: test_oopStorage no longer needs to disable some tests on WIN32
f6c2891: 8267229: Split runtime/Metaspace/elastic test configurations for better scalability
b60975d: 8267237: ARM32: bad AD file in matcher.cpp after 8266810
905b41a: 8265711: C1: Intrinsify Class.getModifier method
... and 149 more: https://git.openjdk.java.net/jdk/compare/c665dba591ae5c15c9ca49e14d1aaa4eea38e7ae...master

Your commit was automatically rebased without conflicts.

Pushed as commit b92c5a4.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Fix java/foreign SIGBUS on macos/aarch64

cb9723b

openjdk bot added the rfr Pull request is ready for review label May 7, 2021

openjdk bot added the hotspot hotspot-dev@openjdk.org label May 7, 2021

theRealAph reviewed May 8, 2021

View reviewed changes

dholmes-ora approved these changes May 10, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label May 10, 2021

Add comments

d46cd67

openjdk bot closed this May 18, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels May 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8265292: [macos_aarch64] java/foreign/TestDowncall.java crashes with SIGBUS #3921

8265292: [macos_aarch64] java/foreign/TestDowncall.java crashes with SIGBUS #3921

AntonKozlov commented May 7, 2021 •

edited by openjdk bot

Loading

bridgekeeper bot commented May 7, 2021

AntonKozlov commented May 7, 2021

openjdk bot commented May 7, 2021

openjdk bot commented May 7, 2021

openjdk bot commented May 7, 2021

mlbridge bot commented May 7, 2021 •

edited

Loading

theRealAph May 8, 2021 •

edited

Loading

theRealAph May 9, 2021

AntonKozlov May 11, 2021

dholmes-ora left a comment

openjdk bot commented May 10, 2021 •

edited

Loading

mlbridge bot commented May 14, 2021

AntonKozlov commented May 17, 2021

theRealAph commented May 18, 2021

AntonKozlov commented May 18, 2021

openjdk bot commented May 18, 2021

		ThreadToNativeFromVM ttnfvm(thread);
		MACOS_AARCH64_ONLY(ThreadWXEnable wx(WXExec, thread));

8265292: [macos_aarch64] java/foreign/TestDowncall.java crashes with SIGBUS #3921

8265292: [macos_aarch64] java/foreign/TestDowncall.java crashes with SIGBUS #3921

Conversation

AntonKozlov commented May 7, 2021 • edited by openjdk bot Loading

Progress

Issues

Reviewers

Reviewing

bridgekeeper bot commented May 7, 2021

AntonKozlov commented May 7, 2021

openjdk bot commented May 7, 2021

openjdk bot commented May 7, 2021

openjdk bot commented May 7, 2021

mlbridge bot commented May 7, 2021 • edited Loading

Webrevs

theRealAph May 8, 2021 • edited Loading

Choose a reason for hiding this comment

theRealAph May 9, 2021

Choose a reason for hiding this comment

AntonKozlov May 11, 2021

Choose a reason for hiding this comment

dholmes-ora left a comment

Choose a reason for hiding this comment

openjdk bot commented May 10, 2021 • edited Loading

mlbridge bot commented May 14, 2021

AntonKozlov commented May 17, 2021

theRealAph commented May 18, 2021

AntonKozlov commented May 18, 2021

openjdk bot commented May 18, 2021

AntonKozlov commented May 7, 2021 •

edited by openjdk bot

Loading

mlbridge bot commented May 7, 2021 •

edited

Loading

theRealAph May 8, 2021 •

edited

Loading

openjdk bot commented May 10, 2021 •

edited

Loading