8277990: NMT: Remove NMT shutdown capability #6640

zhengyu123 · 2021-12-01T16:00:09Z

NMT shutdown functionality is a remnant of its first implementation, which could consume excessive amount of memory, therefore, it needed capability to shut it self down to ensure health of JVM. This is no longer a case for current implementation.

After JDK-8277946, there is no longer a use, so it can be removed.

Test:

hotspot_nmt
tier1 with NMT on

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8277990: NMT: Remove NMT shutdown capability

Reviewers

Thomas Stuefe (@tstuefe - Reviewer) ⚠️ Review applies to 58d2242
Aleksey Shipilev (@shipilev - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/6640/head:pull/6640
$ git checkout pull/6640

Update a local copy of the PR:
$ git checkout pull/6640
$ git pull https://git.openjdk.java.net/jdk pull/6640/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 6640

View PR using the GUI difftool:
$ git pr show -t 6640

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/6640.diff

bridgekeeper · 2021-12-01T16:01:39Z

👋 Welcome back zgu! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-12-01T16:03:42Z

@zhengyu123 The following label will be automatically applied to this pull request:

hotspot

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2021-12-01T16:08:36Z

Webrevs

tstuefe · 2021-12-02T05:09:04Z

Very nice simplification.

Before diving in, could you explain a bit the locking changes in MallocSiteTable? To me, the association with shutdown is not immediately clear. Thanks!

zhengyu123 · 2021-12-02T13:39:02Z

Very nice simplification.

Before diving in, could you explain a bit the locking changes in MallocSiteTable? To me, the association with shutdown is not immediately clear. Thanks!

You are talking about AccessLock, right?

As @dholmes-ora mentioned in PR #6267, the name is misleading: it is not a lock, but a countdown latch. It allows multi-reader to access MallocSiteTable, but once an exclusive access is requested, the requester sets counter to negative number and waits all readers to exit, then no readers and writers are allowed, so that MallocSiteTable can be safely destroyed.

AccessLock was invented to guard MallocSiteTable, as ThreadCritical is too expensive for malloc tracking.

dcubed-ojdk · 2021-12-02T15:21:13Z

Hmmm... sounds like a ReadersWriter lock to me... Any number of Readers and a single
Writer after all the Readers have released... It has surprised me before that HotSpot does
not have such a useful mechanism...

tstuefe · 2021-12-02T15:43:58Z

You are talking about AccessLock, right?

As @dholmes-ora mentioned in PR #6267, the name is misleading: it is not a lock, but a countdown latch. It allows multi-reader to access MallocSiteTable, but once an exclusive access is requested, the requester sets counter to negative number and waits all readers to exit, then no readers and writers are allowed, so that MallocSiteTable can be safely destroyed.

AccessLock was invented to guard MallocSiteTable, as ThreadCritical is too expensive for malloc tracking.

Just to see I get it right, AccessLock is not needed anymore because we don't destroy the MallocSiteTable? We don't need it to guard the table when we add callstacks?

zhengyu123 · 2021-12-02T15:46:36Z

Hmmm... sounds like a ReadersWriter lock to me... Any number of Readers and a single Writer after all the Readers have released... It has surprised me before that HotSpot does not have such a useful mechanism...

Well, the difference is that, it does not have well defined critical section. It depends on other structures/operations for data integrity.

zhengyu123 · 2021-12-02T15:52:24Z

You are talking about AccessLock, right?

As @dholmes-ora mentioned in PR #6267, the name is misleading: it is not a lock, but a countdown latch. It allows multi-reader to access MallocSiteTable, but once an exclusive access is requested, the requester sets counter to negative number and waits all readers to exit, then no readers and writers are allowed, so that MallocSiteTable can be safely destroyed.

AccessLock was invented to guard MallocSiteTable, as ThreadCritical is too expensive for malloc tracking.

Just to see I get it right, AccessLock is not needed anymore because we don't destroy the MallocSiteTable? We don't need it to guard the table when we add callstacks?

Yes. Ironically, adding entry only requires shared access, it uses CAS to ensure table integrity :-) If we don't destroy the table, there is no requester for exclusive access, so it is no longer needed.

tstuefe · 2021-12-02T16:06:08Z

Yes. Ironically, adding entry only requires shared access, it uses CAS to ensure table integrity :-) If we don't destroy the table, there is no requester for exclusive access, so it is no longer needed.

I see. That's nice. Thanks for the clarification.

tstuefe

Nice cleanup.

You missed a few:

test/lib/jdk/test/whitebox/WhiteBox.java, NMTChangeTrackingLevel
test/lib/sun/hotspot/WhiteBox.java, same

Will the removal of WB functions cause incompatibilities with older versions of jtreg (can't imagine it does, just wondering)?

Cheers, Thomas

src/hotspot/os/posix/perfMemory_posix.cpp

src/hotspot/share/services/nmtCommon.hpp

zhengyu123 · 2021-12-02T18:16:20Z

Nice cleanup.

You missed a few:

test/lib/jdk/test/whitebox/WhiteBox.java, NMTChangeTrackingLevel

test/lib/sun/hotspot/WhiteBox.java, same

Out of curious, why have two versions?

Will the removal of WB functions cause incompatibilities with older versions of jtreg (can't imagine it does, just wondering)?

No idea.

Cheers, Thomas

tstuefe · 2021-12-02T18:28:36Z

Nice cleanup.
You missed a few:

test/lib/jdk/test/whitebox/WhiteBox.java, NMTChangeTrackingLevel

test/lib/sun/hotspot/WhiteBox.java, same

Out of curious, why have two versions?

No clue :)

tstuefe

Looks good to me. Thank you for doing this!

..Thomas

openjdk · 2021-12-02T18:31:00Z

@zhengyu123 This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8277990: NMT: Remove NMT shutdown capability

Reviewed-by: stuefe, shade

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 107 new commits pushed to the master branch:

bd7c54a: 8278341: Liveness check for global scope is not as fast as it could be
c609b5d: 8277628: Spec for InetAddressResolverProvider::get() throwing error or exception could be clearer
bb50b92: 8277536: Use String.blank in jdk.javadoc where applicable
5b81d5e: 8276901: Implement UseHeavyMonitors consistently
69d8669: 8278339: ServerSocket::isClosed may return false after accept throws
56ca66e: 8277863: Deprecate sun.misc.Unsafe methods that return offsets
3536127: 8277383: VM.metaspace optionally show chunk freelist details
44fcee3: 8278289: Drop G1BlockOffsetTablePart::_object_can_span
b2638e5: 8244602: Add JTREG_REPEAT_COUNT to repeat execution of a test
07669e3: 8275375: [REDO] JDK-8271949 dumppath in -XX:FlightRecorderOptions does not affect
... and 97 more: https://git.openjdk.java.net/jdk/compare/abaa073bcbdb202658c8a97401ffb098d71e0f16...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

mlbridge · 2021-12-03T03:56:39Z

Mailing list message from David Holmes on hotspot-dev:

On 3/12/2021 4:23 am, Zhengyu Gu wrote:

On Thu, 2 Dec 2021 16:23:32 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

Nice cleanup.

You missed a few:

* test/lib/jdk/test/whitebox/WhiteBox.java, `NMTChangeTrackingLevel`
* test/lib/sun/hotspot/WhiteBox.java, same

Out of curious, why have two versions?

Historical. WB is primarily a hotspot test aid and resided in the
hotspot testlibrary in the hotspot repo. Overtime we've been moving to a
common testlibrary for everyone now we are all in one repo. No idea
where we are with that consolidation process though. I think the person
driving that may no longer be around, not sure.

Will the removal of WB functions cause incompatibilities with older versions of jtreg (can't imagine it does, just wondering)?

If you mean in terms of @require checking that involves WB I believe
that support class is part of the repo not jtreg.

Cheers,
David
-----

zhengyu123 · 2021-12-03T19:27:18Z

Looks good to me. Thank you for doing this!

..Thomas

Thanks for the review, @tstuefe

shipilev

Very nice cleanup. Some minor suggestions below, feel free to ignore them.

shipilev · 2021-12-07T13:41:08Z

src/hotspot/share/services/mallocTracker.cpp

@@ -281,7 +260,7 @@ void* MallocTracker::record_malloc(void* malloc_base, size_t size, MEMFLAGS flag
  assert(((size_t)memblock & (sizeof(size_t) * 2 - 1)) == 0, "Alignment check");

 #ifdef ASSERT
-  if (level > NMT_minimal) {
+  if (level >= NMT_summary) {


Suggested change

if (level >= NMT_summary) {

if (level > NMT_off) {

This way, we don't care where NMT_summary is in the enum. We enter this code on all paths when NMT is enabled?

Actually, maybe even call to enabled() here?

shipilev · 2021-12-07T13:43:19Z

src/hotspot/share/services/memTracker.cpp

-    NMT_TrackingLevel level = tracking_level();
-    if (level >= NMT_summary) {
-      report(level == NMT_summary, output, 1);
+    if (enabled()) {


Looks to me, you can do enabled() && Atomic::cmpxchg(...), thus saving a tiny bit of CPU cycles when NMT is disabled?

shipilev · 2021-12-07T13:44:41Z

src/hotspot/share/services/memTracker.hpp

-  // Transition the tracking level to specified level
-  static bool transition_to(NMT_TrackingLevel level);
+  static inline bool enabled() {
+    return _tracking_level >= NMT_summary;


Suggested change

return _tracking_level >= NMT_summary;

return _tracking_level > NMT_off;

Same reason as above.

tstuefe · 2021-12-07T13:49:57Z

Let's ship this, I'm really interested in getting this into JDK 18 before ramp-down on Thursday.

tstuefe · 2021-12-07T16:10:44Z

Patch is still good after latest changes.

shipilev

Good. Yes, let's try to make it into JDK 18.

zhengyu123 · 2021-12-07T16:31:15Z

Thanks, @tstuefe @shipilev. Will integrate once GHA is clean

zhengyu123 · 2021-12-07T17:41:02Z

/integrate

openjdk · 2021-12-07T17:42:23Z

Going to push as commit 5a036ac.
Since your change was applied there have been 110 commits pushed to the master branch:

7217cb7: 8274883: (se) Selector.open throws IAE when the default file system provider is changed to a custom provider
7ea4b19: 8278166: java/nio/channels/Channels/TransferTo.java timed out
543d1a8: 8275721: Name of UTC timezone in a locale changes depending on previous code
bd7c54a: 8278341: Liveness check for global scope is not as fast as it could be
c609b5d: 8277628: Spec for InetAddressResolverProvider::get() throwing error or exception could be clearer
bb50b92: 8277536: Use String.blank in jdk.javadoc where applicable
5b81d5e: 8276901: Implement UseHeavyMonitors consistently
69d8669: 8278339: ServerSocket::isClosed may return false after accept throws
56ca66e: 8277863: Deprecate sun.misc.Unsafe methods that return offsets
3536127: 8277383: VM.metaspace optionally show chunk freelist details
... and 100 more: https://git.openjdk.java.net/jdk/compare/abaa073bcbdb202658c8a97401ffb098d71e0f16...master

Your commit was automatically rebased without conflicts.

openjdk · 2021-12-07T17:42:46Z

@zhengyu123 Pushed as commit 5a036ac.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Zhengyu Gu added 2 commits December 1, 2021 08:15

v0

83afa64

Fix windows

18415cf

Fix comment

8a5bb2e

openjdk bot added the rfr Pull request is ready for review label Dec 1, 2021

openjdk bot added the hotspot hotspot-dev@openjdk.org label Dec 1, 2021

tstuefe suggested changes Dec 2, 2021

View reviewed changes

src/hotspot/os/posix/perfMemory_posix.cpp Outdated Show resolved Hide resolved

src/hotspot/share/services/nmtCommon.hpp Show resolved Hide resolved

src/hotspot/share/services/nmtCommon.hpp Show resolved Hide resolved

tstuefe's comments

58d2242

tstuefe approved these changes Dec 2, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Dec 2, 2021

shipilev approved these changes Dec 7, 2021

View reviewed changes

Aleksey's comments

30aa1a5

shipilev approved these changes Dec 7, 2021

View reviewed changes

openjdk bot closed this Dec 7, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Dec 7, 2021

	return _tracking_level >= NMT_summary;
	return _tracking_level > NMT_off;

8277990: NMT: Remove NMT shutdown capability #6640

8277990: NMT: Remove NMT shutdown capability #6640

Uh oh!

Conversation

zhengyu123 commented Dec 1, 2021 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Reviewing

Uh oh!

bridgekeeper bot commented Dec 1, 2021

Uh oh!

openjdk bot commented Dec 1, 2021

Uh oh!

mlbridge bot commented Dec 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

tstuefe commented Dec 2, 2021

Uh oh!

zhengyu123 commented Dec 2, 2021

Uh oh!

dcubed-ojdk commented Dec 2, 2021

Uh oh!

tstuefe commented Dec 2, 2021

Uh oh!

zhengyu123 commented Dec 2, 2021

Uh oh!

zhengyu123 commented Dec 2, 2021

Uh oh!

tstuefe commented Dec 2, 2021

Uh oh!

tstuefe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhengyu123 commented Dec 2, 2021

Uh oh!

tstuefe commented Dec 2, 2021

Uh oh!

tstuefe left a comment

Choose a reason for hiding this comment

Uh oh!

openjdk bot commented Dec 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mlbridge bot commented Dec 3, 2021

Uh oh!

zhengyu123 commented Dec 3, 2021

Uh oh!

shipilev left a comment

Choose a reason for hiding this comment

Uh oh!

shipilev Dec 7, 2021

Choose a reason for hiding this comment

Uh oh!

shipilev Dec 7, 2021

Choose a reason for hiding this comment

Uh oh!

shipilev Dec 7, 2021

Choose a reason for hiding this comment

Uh oh!

shipilev Dec 7, 2021

Choose a reason for hiding this comment

Uh oh!

tstuefe commented Dec 7, 2021

Uh oh!

tstuefe commented Dec 7, 2021

Uh oh!

shipilev left a comment

Choose a reason for hiding this comment

Uh oh!

zhengyu123 commented Dec 7, 2021

Uh oh!

zhengyu123 commented Dec 7, 2021

Uh oh!

openjdk bot commented Dec 7, 2021

zhengyu123 commented Dec 1, 2021 •

edited by openjdk bot

Loading

mlbridge bot commented Dec 1, 2021 •

edited

Loading

openjdk bot commented Dec 2, 2021 •

edited

Loading