8320061: [nmt] Multiple issues with peak accounting #481

Delawen · 2024-04-10T08:35:59Z

This is a follow up of #440

This backport fixes several NMT things, among those, the problem with largest_committed being 0KB as shown on the previous PR.

Example of output:

$ jcmd 20088 VM.native_memory detail | grep "mmap: reserved="
                            (mmap: reserved=8388608KB, committed=618496KB, at peak) 
                            (mmap: reserved=1048576KB, committed=32064KB, at peak) 
                            (mmap: reserved=528392KB, committed=41416KB, at peak) 
                            (mmap: reserved=197140KB, committed=45380KB, at peak) 
                            (mmap: reserved=40KB, committed=36KB, at peak) 
                            (mmap: reserved=8KB, committed=8KB, at peak) 
                            (mmap: reserved=262144KB, committed=224640KB, at peak) 
                            (mmap: reserved=0KB, committed=0KB, peak=20KB)

As described in the original PR, other features in this PR is fixing the scale so it also shows when the allocation value is zero (note the GB now instead of showing KB as before, and the 0GB that is shown):

$ jcmd 20088 VM.native_memory detail scale=g | grep "committed"
Total: reserved=11GB, committed=2GB
       mmap:   reserved=10GB, committed=1GB
-                 Java Heap (reserved=8GB, committed=1GB)
                            (mmap: reserved=8GB, committed=1GB, at peak) 
-                     Class (reserved=1GB, committed=0GB)
                            (mmap: reserved=1GB, committed=0GB, at peak) 
                            (    reserved=0GB, committed=0GB)
                            (    reserved=1GB, committed=0GB)
-                      Code (reserved=1GB, committed=0GB)
                            (mmap: reserved=1GB, committed=0GB, at peak) 
-                     Other (reserved=1GB, committed=1GB)
	[0x00000005c0000000 - 0x0000000601c00000] committed 1GB from
                             (reserved=8GB, committed=1GB Type=Java Heap)
                             (reserved=1GB, committed=0GB Type=Class)
                             (reserved=1GB, committed=0GB Type=Code)

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
JDK-8320061 needs maintainer approval
Change must be properly reviewed (2 reviews required, with at least 1 Reviewer, 1 Author)

Issue

JDK-8320061: [nmt] Multiple issues with peak accounting (Enhancement - P4 - Approved)

Reviewers

Aleksey Shipilev (@shipilev - Reviewer)
Thomas Stuefe (@tstuefe - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk21u-dev.git pull/481/head:pull/481
$ git checkout pull/481

Update a local copy of the PR:
$ git checkout pull/481
$ git pull https://git.openjdk.org/jdk21u-dev.git pull/481/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 481

View PR using the GUI difftool:
$ git pr show -t 481

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk21u-dev/pull/481.diff

Webrev

Link to Webrev Comment

bridgekeeper · 2024-04-10T08:36:58Z

👋 Welcome back Delawen! A progress list of the required criteria for merging this PR into pr/440 will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2024-04-10T08:37:48Z

@Delawen This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8320061: [nmt] Multiple issues with peak accounting

Reviewed-by: shade, stuefe

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 8 new commits pushed to the master branch:

1d01cd7: 8315889: Open source several Swing HTMLDocument related tests
2331bc7: 8329013: StackOverflowError when starting Apache Tomcat with signed jar
68d0925: 8324933: ConcurrentHashTable::statistics_calculate synchronization is expensive
2626d73: 8327989: java/net/httpclient/ManyRequest.java should not use "localhost" in URIs
9c0e18c: 8320924: Improve heap dump performance by optimizing archived object checks
41237fc: 8320707: Virtual thread test updates
955fcc1: 8310355: Move the stub test from initialize_final_stubs() to test/hotspot/gtest
031acf1: 8327096: (fc) java/nio/channels/FileChannel/Size.java fails on partition incapable of creating large files

Please see this link for an up-to-date comparison between the source branch of this pull request and the master branch.
As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@shipilev, @tstuefe) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

openjdk · 2024-04-10T08:38:09Z

This backport pull request has now been updated with issue from the original commit.

mlbridge · 2024-04-10T08:41:30Z

Webrevs

Delawen · 2024-04-10T08:50:21Z

/clean

This is a clean merge to another open PR.

openjdk · 2024-04-10T08:50:41Z

@Delawen Only OpenJDK Committers can use the /clean command

openjdk-notifier · 2024-04-23T10:58:08Z

The parent pull request that this pull request depends on has now been integrated and the target branch of this pull request has been updated. This means that changes from the dependent pull request can start to show up as belonging to this pull request, which may be confusing for reviewers. To remedy this situation, simply merge the latest changes from the new target branch into this pull request by running commands similar to these in the local repository for your personal fork:

git checkout backport-JDK-8320061
git fetch https://git.openjdk.org/jdk21u-dev.git master
git merge FETCH_HEAD
# if there are conflicts, follow the instructions given by git merge
git commit -m "Merge master"
git push

shipilev

Superficially, this looks good. @tstuefe should ack as well.

Have you ran runtime/NMT tests with this PR?

shipilev · 2024-04-23T11:36:00Z

/reviewers 2

openjdk · 2024-04-23T11:36:40Z

⚠️ @Delawen This change is now ready for you to apply for maintainer approval. This can be done directly in each associated issue or by using the /approval command.

openjdk · 2024-04-23T11:36:47Z

@shipilev
The total number of required reviews for this PR (including the jcheck configuration and the last /reviewers command) is now set to 2 (with at least 1 Reviewer, 1 Author).

Delawen · 2024-04-23T11:40:53Z

Tests are being run on the github jobs right now. Once they are done I will do the approval command. Thanks @shipilev !

shipilev · 2024-04-23T11:46:21Z

Tests are being run on the github jobs right now. Once they are done I will do the approval command. Thanks @shipilev !

It would be safer to run runtime/NMT locally, without relying on GHA to include all of them. Since this PR is modifies the tests, it is expected that you verify all of them still work reliably :)

Delawen · 2024-04-23T11:50:36Z

Sure, then I will run them now :)

(I did it on the original PR commit but it's true, a merge that big deserves another try)

Delawen · 2024-04-23T14:21:26Z

Tests passed.

Also I checked again the output of the following commands:

$ jcmd 2847003 VM.native_memory detail | grep "mmap: reserved="
                            (mmap: reserved=8388608KB, committed=1236992KB, at peak) 
                            (mmap: reserved=1048576KB, committed=40384KB, at peak) 
                            (mmap: reserved=528392KB, committed=68440KB, at peak) 
                            (mmap: reserved=197140KB, committed=57460KB, at peak) 
                            (mmap: reserved=40KB, committed=36KB, at peak) 
                            (mmap: reserved=8KB, committed=8KB, at peak) 
                            (mmap: reserved=327680KB, committed=276224KB, at peak) 
                            (mmap: reserved=0KB, committed=0KB, peak=20KB)

$ jcmd 2847003 VM.native_memory detail scale=g | grep "committed"
Total: reserved=11GB, committed=3GB
       mmap:   reserved=10GB, committed=2GB
-                 Java Heap (reserved=8GB, committed=1GB)
                            (mmap: reserved=8GB, committed=1GB, at peak) 
-                     Class (reserved=1GB, committed=0GB)
                            (mmap: reserved=1GB, committed=0GB, at peak) 
                            (    reserved=0GB, committed=0GB)
                            (    reserved=1GB, committed=0GB)
-                      Code (reserved=1GB, committed=0GB)
                            (mmap: reserved=1GB, committed=0GB, at peak) 
-                     Other (reserved=1GB, committed=1GB)
	[0x00000005c0000000 - 0x000000060c800000] committed 1GB from
                             (reserved=8GB, committed=1GB Type=Java Heap)
                             (reserved=1GB, committed=0GB Type=Class)
                             (reserved=1GB, committed=0GB Type=Code)

Delawen · 2024-04-23T14:23:09Z

/approval JDK-8320061 request Fix Request
This is a backport for https://bugs.openjdk.org/browse/JDK-8320061 ( [nmt] Multiple issues with peak accounting ). This backport fixes several NMT things, among those, the problem with largest_committed being 0KB as shown on the previous PR.

openjdk · 2024-04-23T14:23:45Z

@Delawen
JDK-8320061: The approval request has been created successfully.

Delawen · 2024-04-24T09:54:35Z

@tstuefe can you take a look here?

tstuefe · 2024-04-26T09:15:58Z

Hi @Delawen,

thank you for doing this. Can you describe what parts of the patch you had to adapt, in order to make my review easier? Were the needed changes trivial, e.g. only comments?

Delawen · 2024-04-26T09:26:52Z

I didn't have to change much, just fixing a couple of trivial merge conflicts (line numbers changed and that made git ask if it was right).

It is mostly the original backport. If you compare the original commit and this one you will see the changes shown are the same.

tstuefe

Thank you!

tstuefe · 2024-04-26T09:41:26Z

I didn't have to change much, just fixing a couple of trivial merge conflicts (line numbers changed and that made git ask if it was right).

It is mostly the original backport. If you compare the original commit and this one you will see the changes shown are the same.

Thanks, looks good. I wish we had a tool for showing the manual changes after such a merge.

Delawen · 2024-04-26T10:03:22Z

/integrate

openjdk · 2024-04-26T10:05:36Z

@Delawen
Your change (at version 77ac5fe) is now ready to be sponsored by a Committer.

shipilev · 2024-04-26T11:11:48Z

Thanks both! Let's do this.

/sponsor

openjdk · 2024-04-26T11:14:52Z

Going to push as commit 262cacb.
Since your change was applied there have been 8 commits pushed to the master branch:

1d01cd7: 8315889: Open source several Swing HTMLDocument related tests
2331bc7: 8329013: StackOverflowError when starting Apache Tomcat with signed jar
68d0925: 8324933: ConcurrentHashTable::statistics_calculate synchronization is expensive
2626d73: 8327989: java/net/httpclient/ManyRequest.java should not use "localhost" in URIs
9c0e18c: 8320924: Improve heap dump performance by optimizing archived object checks
41237fc: 8320707: Virtual thread test updates
955fcc1: 8310355: Move the stub test from initialize_final_stubs() to test/hotspot/gtest
031acf1: 8327096: (fc) java/nio/channels/FileChannel/Size.java fails on partition incapable of creating large files

Your commit was automatically rebased without conflicts.

openjdk · 2024-04-26T11:15:03Z

@shipilev @Delawen Pushed as commit 262cacb.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Delawen added 2 commits April 3, 2024 11:14

Backport 8647f001bbb804503e8491d10f77aa0f34d66bf2

57e50fe

Backport dc256fbc6490f8163adb286dbb7380c10e5e1e06

462bec0

openjdk bot changed the title ~~Backport dc256fbc6490f8163adb286dbb7380c10e5e1e06~~ 8320061: [nmt] Multiple issues with peak accounting Apr 10, 2024

openjdk bot added backport rfr Pull request is ready for review labels Apr 10, 2024

jerboaa mentioned this pull request Apr 10, 2024

8293850: need a largest_committed metric for each category of NMT's output #440

Closed

3 tasks

openjdk-notifier bot changed the base branch from pr/440 to master April 23, 2024 10:58

Merge master

77ac5fe

shipilev approved these changes Apr 23, 2024

View reviewed changes

openjdk bot added approval and removed approval labels Apr 23, 2024

tstuefe approved these changes Apr 26, 2024

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Apr 26, 2024

openjdk bot added the sponsor Pull request is ready to be sponsored label Apr 26, 2024

openjdk bot added the integrated Pull request has been integrated label Apr 26, 2024

openjdk bot closed this Apr 26, 2024

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review sponsor Pull request is ready to be sponsored labels Apr 26, 2024

Delawen deleted the backport-JDK-8320061 branch April 26, 2024 11:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8320061: [nmt] Multiple issues with peak accounting #481

8320061: [nmt] Multiple issues with peak accounting #481

Delawen commented Apr 10, 2024 •

edited by openjdk bot

Loading

bridgekeeper bot commented Apr 10, 2024

openjdk bot commented Apr 10, 2024 •

edited

Loading

openjdk bot commented Apr 10, 2024

mlbridge bot commented Apr 10, 2024 •

edited

Loading

Delawen commented Apr 10, 2024

openjdk bot commented Apr 10, 2024

openjdk-notifier bot commented Apr 23, 2024

shipilev left a comment

shipilev commented Apr 23, 2024

openjdk bot commented Apr 23, 2024

openjdk bot commented Apr 23, 2024

Delawen commented Apr 23, 2024

shipilev commented Apr 23, 2024

Delawen commented Apr 23, 2024

Delawen commented Apr 23, 2024 •

edited

Loading

Delawen commented Apr 23, 2024

openjdk bot commented Apr 23, 2024

Delawen commented Apr 24, 2024

tstuefe commented Apr 26, 2024

Delawen commented Apr 26, 2024

tstuefe left a comment

tstuefe commented Apr 26, 2024

Delawen commented Apr 26, 2024

openjdk bot commented Apr 26, 2024

shipilev commented Apr 26, 2024

openjdk bot commented Apr 26, 2024

openjdk bot commented Apr 26, 2024

8320061: [nmt] Multiple issues with peak accounting #481

8320061: [nmt] Multiple issues with peak accounting #481

Conversation

Delawen commented Apr 10, 2024 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Reviewing

Webrev

bridgekeeper bot commented Apr 10, 2024

openjdk bot commented Apr 10, 2024 • edited Loading

openjdk bot commented Apr 10, 2024

mlbridge bot commented Apr 10, 2024 • edited Loading

Webrevs

Delawen commented Apr 10, 2024

openjdk bot commented Apr 10, 2024

openjdk-notifier bot commented Apr 23, 2024

shipilev left a comment

Choose a reason for hiding this comment

shipilev commented Apr 23, 2024

openjdk bot commented Apr 23, 2024

openjdk bot commented Apr 23, 2024

Delawen commented Apr 23, 2024

shipilev commented Apr 23, 2024

Delawen commented Apr 23, 2024

Delawen commented Apr 23, 2024 • edited Loading

Delawen commented Apr 23, 2024

openjdk bot commented Apr 23, 2024

Delawen commented Apr 24, 2024

tstuefe commented Apr 26, 2024

Delawen commented Apr 26, 2024

tstuefe left a comment

Choose a reason for hiding this comment

tstuefe commented Apr 26, 2024

Delawen commented Apr 26, 2024

openjdk bot commented Apr 26, 2024

shipilev commented Apr 26, 2024

openjdk bot commented Apr 26, 2024

openjdk bot commented Apr 26, 2024

Delawen commented Apr 10, 2024 •

edited by openjdk bot

Loading

openjdk bot commented Apr 10, 2024 •

edited

Loading

mlbridge bot commented Apr 10, 2024 •

edited

Loading

Delawen commented Apr 23, 2024 •

edited

Loading