8253857: Shenandoah: Bugs in ShenandoahEvacOOMHandler related code

[zhengyu123]

There are a couple of bugs in ShenandoahEvacOOMHandler related code.

1. ShenandoahEvacOOMHandler::register_thread()
   It does not do OOM check before blindly cmpxchg in threads_in_evac. The cmpxchg can success even OOM is in progress, that results the thread to continue evacuate objects.

2. When a nested OOM scope encounters OOM, it does not decrease threads_in_evac count before entering wait_for_no_evac_threads() loop, it deadlocks itself.

Test:

• hotspot_gc_shenandoah on Linux x86_64

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed

Testing

	Linux x64	Windows x64	macOS x64
Build	✔️ (3/3 passed)	✔️ (2/2 passed)	✔️ (2/2 passed)
Test (tier1)	✔️ (9/9 passed)	✔️ (9/9 passed)	✔️ (9/9 passed)

Issue

• JDK-8253857: Shenandoah: Bugs in ShenandoahEvacOOMHandler related code

Reviewers

• Roman Kennke (@rkennke - Reviewer)

Download

$ git fetch https://git.openjdk.java.net/jdk pull/439/head:pull/439
$ git checkout pull/439

[bridgekeeper]

👋 Welcome back zgu! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@zhengyu123 The following labels will be automatically applied to this pull request:

• hotspot-gc
• shenandoah

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (a1cae46)

[rkennke]

Looks good! Thanks!

[openjdk]

@zhengyu123 This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8253857: Shenandoah: Bugs in ShenandoahEvacOOMHandler related code

Reviewed-by: rkennke

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 130 new commits pushed to the master branch:

• e9c1905: 8253740: [PPC64] Minor interpreter cleanup
• b1448da: 8253900: SA: wrong size computation when JVM was built without AOT
• 2bc8bc5: 8254265: s390 and linux 32 bit builds broken
• 4f9a1ff: 8254073: Tokenizer improvements (revised)
• 9cecc16: 8254244: Some code emitted by TemplateTable::branch is unused when running TieredCompilation
• a95590d: 8254285: G1: Remove "What is this about" comment in G1CollectedHeap.cpp
• 0230781: 8254175: Build no-pch configuration in debug mode for submit checks
• b9873e1: 8253180: ZGC: Implementation of JEP 376: ZGC: Concurrent Thread-Stack Processing
• a2f6519: 8233685: Test tools/javac/modules/AddLimitMods.java fails
• 70be8c7: 8253965: Delete the outdated java.awt.PeerFixer class
• ... and 120 more: https://git.openjdk.java.net/jdk/compare/8cf8e463c623b3908befb0af289749b47a793267...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[zhengyu123]

/integrate

[openjdk]

@zhengyu123 Since your change was applied there have been 130 commits pushed to the master branch:

• e9c1905: 8253740: [PPC64] Minor interpreter cleanup
• b1448da: 8253900: SA: wrong size computation when JVM was built without AOT
• 2bc8bc5: 8254265: s390 and linux 32 bit builds broken
• 4f9a1ff: 8254073: Tokenizer improvements (revised)
• 9cecc16: 8254244: Some code emitted by TemplateTable::branch is unused when running TieredCompilation
• a95590d: 8254285: G1: Remove "What is this about" comment in G1CollectedHeap.cpp
• 0230781: 8254175: Build no-pch configuration in debug mode for submit checks
• b9873e1: 8253180: ZGC: Implementation of JEP 376: ZGC: Concurrent Thread-Stack Processing
• a2f6519: 8233685: Test tools/javac/modules/AddLimitMods.java fails
• 70be8c7: 8253965: Delete the outdated java.awt.PeerFixer class
• ... and 120 more: https://git.openjdk.java.net/jdk/compare/8cf8e463c623b3908befb0af289749b47a793267...master

Your commit was automatically rebased without conflicts.

Pushed as commit d4b5dfd.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.