Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JDK-8275582: Don't purge metaspace mapping lists #6036

Closed

Conversation

tstuefe
Copy link
Member

@tstuefe tstuefe commented Oct 20, 2021

Metaspace, at its lowest level, keeps a list of memory mappings. That list is dynamically expandable, which makes Metaspace expandable as well, one of the key features of this allocator.

There are two mechanisms in place in Metaspace to return memory to the OS:

  1. purging the list of mappings
  2. uncommitting free chunks

(1) is the old way, existing since the inception of Metaspace. It works by unmapping and deleting mapping segments which are fully evacuated. Metaspace allocations are fine granular and cannot move, so Metaspace is subject to fragmentation. That fragmentation makes (1) rather ineffective since we only can unmap a mapping that had been fully free'd after class unloading. The mappings however are quite large (and we plan to make them larger still), the chance of one being completely empty is small.

(2) was introduced with JEP-387 and is much more effective. We return memory not on a mapping level, but on a chunk level, by uncommitting free chunks larger than a certain threshold. That works very well.

When working on JEP-387, I left (1) in place out of caution. But (1) is really not needed anymore. It has no effect in combination with (2) because if a whole mapping is free and eligible for unmapping using technique(1), all its chunks must be free, will have been maximally folded by the buddy allocator, and their backing memory will have been completely uncommitted. So unmapping that segment has no effect on the process' RSS.

Removing (1) would have the following advantages:

  • a nice reduction in code complexity
  • reduce the work we do during class unloading
  • the mapping list then would only ever grow, never shrink, which would simplify lockless accesses (see JDK-8271124)
  • (1) stands in the way of other improvements, e.g. the increase of the mapping size of individual mappings, which would have diminished the effectiveness of (1) even further.

Removing the ability to purge the mapping list means we retain the uncommitted virtual address space of those mappings after spikes in class unloading. But the consequences of that are minimal, it does not affect RSS at all.


This patch removes old-style metaspace node purging completely. Therefore it's almost all line deletions.

Tests:

  • manual, x64 + x86 Linux, tested all metaspace related jtreg tests
  • manually tested that we still reclaim memory as we did before
  • GHAs
  • SAP nightly tests

Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed

Issue

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/6036/head:pull/6036
$ git checkout pull/6036

Update a local copy of the PR:
$ git checkout pull/6036
$ git pull https://git.openjdk.java.net/jdk pull/6036/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 6036

View PR using the GUI difftool:
$ git pr show -t 6036

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/6036.diff

@bridgekeeper
Copy link

@bridgekeeper bridgekeeper bot commented Oct 20, 2021

👋 Welcome back stuefe! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk
Copy link

@openjdk openjdk bot commented Oct 20, 2021

@tstuefe The following label will be automatically applied to this pull request:

  • hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added the hotspot-runtime label Oct 20, 2021
@tstuefe tstuefe force-pushed the JDK-8275582-dont-purge-metaspace branch from 1cbe0a0 to 10ad2af Compare Oct 21, 2021
@tstuefe tstuefe marked this pull request as ready for review Oct 22, 2021
@openjdk openjdk bot added the rfr label Oct 22, 2021
@mlbridge
Copy link

@mlbridge mlbridge bot commented Oct 22, 2021

Webrevs

Copy link
Contributor

@coleenp coleenp left a comment

Looks great!

@openjdk
Copy link

@openjdk openjdk bot commented Oct 25, 2021

@tstuefe This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8275582: Don't purge metaspace mapping lists

Reviewed-by: coleenp, lkorinth

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 106 new commits pushed to the master branch:

  • 337a9b7: 8269853: Prefetch::read should accept pointer to const
  • 97d3280: 8275536: Add test to check that File::lastModified returns same time stamp as Files.getLastModifiedTime
  • 89671aa: 8273712: C2: Add mechanism for rejecting inlining of low frequency call sites and deprecate MinInliningThreshold.
  • 3221a14: 8273678: TableAccessibility and TableRowAccessibility miss autorelease
  • 7cf68b1: 8202932: java/awt/Component/NativeInLightShow/NativeInLightShow.java fails
  • f610ef0: 8196440: Regression automated Test 'java/awt/TrayIcon/PopupMenuLeakTest/PopupMenuLeakTest.java' fails
  • f143d2a: 8268595: java/io/Serializable/serialFilter/GlobalFilterTest.java#id1 failed in timeout
  • 7f94302: 8275511: G1: Rename needs_remset_update to remset_is_tracked in G1HeapRegionAttr
  • 0bcc174: 8275717: Reimplement STATIC_ASSERT to use static_assert
  • f623298: 8271199: Mutual TLS handshake fails signing client certificate with custom sensitive PKCS11 key
  • ... and 96 more: https://git.openjdk.java.net/jdk/compare/947d52c4c3deec1bdea43959c200201c614ae114...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready label Oct 25, 2021
Copy link
Contributor

@lkorinth lkorinth left a comment

Looks good to me. Much simpler this way. Thanks for the great description of the change!

@tstuefe
Copy link
Member Author

@tstuefe tstuefe commented Oct 26, 2021

Thanks @coleenp and @lkorinth for the reviews!

/integrate

@openjdk
Copy link

@openjdk openjdk bot commented Oct 26, 2021

Going to push as commit 3ff085e.
Since your change was applied there have been 108 commits pushed to the master branch:

  • 10e1610: 8251134: Unwrapping a key with a Private Key generated by Microsoft CNG fails
  • 4361945: 8185844: MSCAPI doesn't list aliases correctly
  • 337a9b7: 8269853: Prefetch::read should accept pointer to const
  • 97d3280: 8275536: Add test to check that File::lastModified returns same time stamp as Files.getLastModifiedTime
  • 89671aa: 8273712: C2: Add mechanism for rejecting inlining of low frequency call sites and deprecate MinInliningThreshold.
  • 3221a14: 8273678: TableAccessibility and TableRowAccessibility miss autorelease
  • 7cf68b1: 8202932: java/awt/Component/NativeInLightShow/NativeInLightShow.java fails
  • f610ef0: 8196440: Regression automated Test 'java/awt/TrayIcon/PopupMenuLeakTest/PopupMenuLeakTest.java' fails
  • f143d2a: 8268595: java/io/Serializable/serialFilter/GlobalFilterTest.java#id1 failed in timeout
  • 7f94302: 8275511: G1: Rename needs_remset_update to remset_is_tracked in G1HeapRegionAttr
  • ... and 98 more: https://git.openjdk.java.net/jdk/compare/947d52c4c3deec1bdea43959c200201c614ae114...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot closed this Oct 26, 2021
@openjdk openjdk bot added integrated and removed ready rfr labels Oct 26, 2021
@openjdk
Copy link

@openjdk openjdk bot commented Oct 26, 2021

@tstuefe Pushed as commit 3ff085e.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

@tstuefe tstuefe deleted the JDK-8275582-dont-purge-metaspace branch Oct 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hotspot-runtime integrated
3 participants