Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JDK-8286198: [linux] Fix process-memory information #8553

Conversation

tstuefe
Copy link
Member

@tstuefe tstuefe commented May 5, 2022

We use mallinfo/mallinfo2 to obtain information about the glibc used memory, but there is an error which causes it to be correct only for fine granular malloc allocations. glibc puts larger allocations in mmap-allocated areas, which were missing from the old calculation.

This patch fixes that; it also adds information about memory retained in the glibc after free. Note that the latter unfortunately does not follow glibc trims because trimming is done AFAIK with madvice(MADV_DONTNEED) which does not seem to affect glibc internal bookkeeping. Still its a very valuable thing to know in order to estimate glibc overhead especially in fine granular mass allocation scenarios.

Tests: Lots of manual tests with various malloc loads and granularities, in addition to automated tests.


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed (1 review required, with at least 1 reviewer)

Issue

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/8553/head:pull/8553
$ git checkout pull/8553

Update a local copy of the PR:
$ git checkout pull/8553
$ git pull https://git.openjdk.java.net/jdk pull/8553/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 8553

View PR using the GUI difftool:
$ git pr show -t 8553

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/8553.diff

@bridgekeeper
Copy link

bridgekeeper bot commented May 5, 2022

👋 Welcome back stuefe! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@tstuefe tstuefe marked this pull request as ready for review May 5, 2022
@openjdk
Copy link

openjdk bot commented May 5, 2022

@tstuefe The following label will be automatically applied to this pull request:

  • hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added hotspot-runtime rfr labels May 5, 2022
@mlbridge
Copy link

mlbridge bot commented May 5, 2022

Webrevs

@tstuefe
Copy link
Member Author

tstuefe commented May 8, 2022

GHA test error on Windows irrelevant for this patch

Copy link
Member

@dholmes-ora dholmes-ora left a comment

I don't understand your "free retained" terminology, but otherwise the changes seem fine and do what you describe.

Thanks,
David

@openjdk
Copy link

openjdk bot commented May 9, 2022

@tstuefe This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8286198: [linux] Fix process-memory information

Reviewed-by: dholmes, mbaesken

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 71 new commits pushed to the master branch:

  • 65f5067: 8286429: jpackageapplauncher build fails intermittently in Tier[45]
  • c4bd449: 8238373: Punctuation should be same in jlink help usage on Japanese language
  • 3fa1c40: 8286298: Remove unused methods in sun.invoke.util.VerifyType
  • 3462190: 8286163: micro-optimize Instant.plusSeconds
  • 60a91d1: 8286285: G1: Rank issues with ParGCRareEvent_lock and Threads_lock
  • d478958: 8286179: Node::find(int) should not traverse from new to old nodes
  • de8f4d0: 8286191: misc tests fail due to JDK-8285987
  • bf0dc4f: 8286367: riscv: riscv port is broken after JDK-8284161
  • 4fd79a6: 8285730: unify _WIN32_WINNT settings
  • bd6026c: 7124282: [macosx] Can't see table cell highlighter when the highlight border is the same color as the cell.
  • ... and 61 more: https://git.openjdk.java.net/jdk/compare/1bb4de2e2868a539846ec48dd43fd623c2ba69a5...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready label May 9, 2022
@tstuefe
Copy link
Member Author

tstuefe commented May 9, 2022

I don't understand your "free retained" terminology, but otherwise the changes seem fine and do what you describe.

Thanks, David

Thank you, David.

Free Retained memory is memory you free()'d but which is retained by the glibc, not handed back to the Operating System.

@MBaesken
Copy link
Member

MBaesken commented May 10, 2022

Hi Thomas, when looking at the manpage https://man7.org/linux/man-pages/man3/mallinfo.3.html
then the manpage is describing "Total allocated space" as just uordblks , see the output in the example program of the manpage.
So maybe it would be better to print mi.uordblks and mi.hblkhd separately so that someone looking at the mallinfo manpage finds exactly the same terminology described there in the JVM output? Maybe in some situations it is useful too to get both values ?

@tstuefe
Copy link
Member Author

tstuefe commented May 10, 2022

Hi Thomas, when looking at the manpage https://man7.org/linux/man-pages/man3/mallinfo.3.html then the manpage is describing "Total allocated space" as just uordblks , see the output in the example program of the manpage. So maybe it would be better to print mi.uordblks and mi.hblkhd separately so that someone looking at the mallinfo manpage finds exactly the same terminology described there in the JVM output? Maybe in some situations it is useful too to get both values ?

No, it's wrong. Hence this patch (see patch description - the number does not account for large areas allocated by the glibc with mmap). Not sure what the state of that manpage is. But I tested this very thoroughly.

Observe:

1a) Old VM allocates 100mio 16 byte blocks:

thomas@starfish:~$ jjjcmd de.stuefe.repros.AllocCHeap VM.info | grep C-Heap
C-Heap outstanding allocations: 3128240K

OK.

1b) Old VM allocates 16 100mio byte blocks:

thomas@starfish:~$ jjjcmd de.stuefe.repros.AllocCHeap VM.info | grep C-Heap
C-Heap outstanding allocations: 5782K

Not OK. It says 5,7M, but we just allocated (payloadsize) ~1.6G.

2a) Patched VM allocates 100mio 16 byte blocks:

thomas@starfish:~$ jjjcmd de.stuefe.repros.AllocCHeap VM.info | grep C-Heap
C-Heap outstanding allocations: 3171375K, retained: 6440K

Still OK.

2b) Patched VM allocates 16 100mio byte blocks:

thomas@starfish:~$ jjjcmd de.stuefe.repros.AllocCHeap VM.info | grep C-Heap
C-Heap outstanding allocations: 1611294K, retained: 2553K

Now it looks legit too, compared to (1b). The lower size (1.6G vs 3.1G) stems from the massively lower glibc overhead when allocating large areas instead of tiny blocks.

@tstuefe
Copy link
Member Author

tstuefe commented May 10, 2022

Thanks @dholmes-ora and @MBaesken !

/integrate

@openjdk
Copy link

openjdk bot commented May 10, 2022

Going to push as commit 9e320d9.
Since your change was applied there have been 71 commits pushed to the master branch:

  • 65f5067: 8286429: jpackageapplauncher build fails intermittently in Tier[45]
  • c4bd449: 8238373: Punctuation should be same in jlink help usage on Japanese language
  • 3fa1c40: 8286298: Remove unused methods in sun.invoke.util.VerifyType
  • 3462190: 8286163: micro-optimize Instant.plusSeconds
  • 60a91d1: 8286285: G1: Rank issues with ParGCRareEvent_lock and Threads_lock
  • d478958: 8286179: Node::find(int) should not traverse from new to old nodes
  • de8f4d0: 8286191: misc tests fail due to JDK-8285987
  • bf0dc4f: 8286367: riscv: riscv port is broken after JDK-8284161
  • 4fd79a6: 8285730: unify _WIN32_WINNT settings
  • bd6026c: 7124282: [macosx] Can't see table cell highlighter when the highlight border is the same color as the cell.
  • ... and 61 more: https://git.openjdk.java.net/jdk/compare/1bb4de2e2868a539846ec48dd43fd623c2ba69a5...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot added the integrated label May 10, 2022
@openjdk openjdk bot closed this May 10, 2022
@openjdk openjdk bot removed ready rfr labels May 10, 2022
@openjdk
Copy link

openjdk bot commented May 10, 2022

@tstuefe Pushed as commit 9e320d9.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hotspot-runtime integrated
3 participants