8259063: Possible deadlock with vtable/itable creation vs concurrent class unloading

[fisk]

There is an offender of consistent lock ordering: the creation of itable/vtable stubs. It happens when an inline cache is transitioning to megamorphic. Way earlier in the call stack, we grab the nmethod lock for IC patching, and then when a megamorphic transition happens and there is no stub created yet for the given vtable/itable index (rather uncommon), a new vtable/itable stub is created. This creation of a stub takes the CodeCache lock. We already deal with failure to create the stubs today, by making the inline cache "clean" instead, essentially deferring the transition to megamorphic to a subsequent call. We deal with that today to handle the code cache running out of memory, making it temporarily impossible to transition to megamorphic until memory has been freed up. My patch rides on that logic, so that we can try_lock() the CodeCache_lock instead. If we fail to take the lock, then it is arguably a bad time to do this megamorphic transition, regardless of deadlocks, as we might have to wait quite a while. So if we fail to take the lock, I just return NULL saying we couldn't create a vtable/itable stub at this time, signalling we defer the transition to later, as we would also do if we couldn't create the stub due to memory exhaustion. This solves the deadlock situation.

Eric Caspole who reported the bug has tried this patch, and it solved the problem. I also tried it with the same test myself, with the same successful results. I also ran it through tier1-5 as it is touching inline cache code which can be a bit subtle.

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed

Issue

• JDK-8259063: Possible deadlock with vtable/itable creation vs concurrent class unloading

Reviewers

• Per Liden (@pliden - Reviewer) ⚠️ Review applies to 71c0979
• Nils Eliasson (@neliasso - Reviewer)

Download

$ git fetch https://git.openjdk.java.net/jdk16 pull/103/head:pull/103
$ git checkout pull/103

[bridgekeeper]

👋 Welcome back eosterlund! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@fisk The following label will be automatically applied to this pull request:

• hotspot-compiler

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (bdf75c7)
• 00: Full (71c0979)

[pliden]

Looks good to me!

[openjdk]

@fisk This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8259063: Possible deadlock with vtable/itable creation vs concurrent class unloading

Reviewed-by: pliden, neliasso

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 6 new commits pushed to the master branch:

• 67e1b63: 8259380: Correct pretouch chunk size to cap with actual page size
• 28ff2de: 8259237: Demo selection changes with left/right arrow key. No need to press space for selection.
• a7e5da2: 8258384: AArch64: SVE verify_ptrue fails on some tests
• 2cb271e: 8253996: Javac error on jdk16 build 18: invalid flag: -Xdoclint:-missing
• d60a937: 8259028: ClassCastException when using custom filesystem with wrapper FileChannel impl
• e05f36f: 8259043: More Zero architectures need linkage with libatomic

Please see this link for an up-to-date comparison between the source branch of this pull request and the master branch.
As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[neliasso]

I like the solution - simple and without adding extra complexity or lock juggling.

It's a good just skipping the call to handle_full_code_cache. The VTableBlobs are not the thing usually filling up the code cache - and they can be placed in any codeheap. And even if we skip it - no harm done - there will be other allocations that will trigger the call.

[fisk]

Thanks for the reviews @neliasso and @pliden.

[fisk]

/integrate

[openjdk]

@fisk Since your change was applied there have been 18 commits pushed to the master branch:

• 6bb6093: 8259657: typo in generated HELP page prevents localization
• 5567530: 8258272: LoadVectorMaskedNode can't be replaced by zero con
• a99df45: 8259560: Zero m68k: "static assertion failed: align" after JDK-8252049
• efc36be: 8258985: Parallel WeakProcessor may use too few threads
• 417e1d1: 8259061: C2: assert(found) failed: memory-writing node is not placed in its original loop or an ancestor of it
• 793017d: 8259601: AArch64: Fix reinterpretX2D match rule issue
• 15dd8f3: 8259275: JRuby crashes while resolving invokedynamic instruction
• 1cf2378: 8259353: VectorReinterpretNode is incorrectly optimized out
• 17b4db3: 8259636: Check for buffer backed by shared segment kicks in in unexpected places
• 5f9cd72: 8259645: Revert JDK-8245956 JavaCompiler still uses File API instead of Path API in a specific case
• ... and 8 more: https://git.openjdk.java.net/jdk16/compare/020ec8485251698d1187204ac13321f4726e45ea...master

Your commit was automatically rebased without conflicts.

Pushed as commit 42d2d6d.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.