JDK-8286198: [linux] Fix process-memory information

[tstuefe]

We use mallinfo/mallinfo2 to obtain information about the glibc used memory, but there is an error which causes it to be correct only for fine granular malloc allocations. glibc puts larger allocations in mmap-allocated areas, which were missing from the old calculation.

This patch fixes that; it also adds information about memory retained in the glibc after free. Note that the latter unfortunately does not follow glibc trims because trimming is done AFAIK with madvice(MADV_DONTNEED) which does not seem to affect glibc internal bookkeeping. Still its a very valuable thing to know in order to estimate glibc overhead especially in fine granular mass allocation scenarios.

Tests: Lots of manual tests with various malloc loads and granularities, in addition to automated tests.

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed (1 review required, with at least 1 reviewer)

Issue

• JDK-8286198: [linux] Fix process-memory information

Reviewers

• David Holmes (@dholmes-ora - Reviewer)
• Matthias Baesken (@MBaesken - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/8553/head:pull/8553
$ git checkout pull/8553

Update a local copy of the PR:
$ git checkout pull/8553
$ git pull https://git.openjdk.java.net/jdk pull/8553/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 8553

View PR using the GUI difftool:
$ git pr show -t 8553

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/8553.diff

[bridgekeeper]

👋 Welcome back stuefe! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@tstuefe The following label will be automatically applied to this pull request:

• hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (7a34e4ad)

[tstuefe]

GHA test error on Windows irrelevant for this patch

[dholmes-ora]

I don't understand your "free retained" terminology, but otherwise the changes seem fine and do what you describe.

Thanks,
David

[openjdk]

@tstuefe This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8286198: [linux] Fix process-memory information

Reviewed-by: dholmes, mbaesken

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 71 new commits pushed to the master branch:

• 65f5067: 8286429: jpackageapplauncher build fails intermittently in Tier[45]
• c4bd449: 8238373: Punctuation should be same in jlink help usage on Japanese language
• 3fa1c40: 8286298: Remove unused methods in sun.invoke.util.VerifyType
• 3462190: 8286163: micro-optimize Instant.plusSeconds
• 60a91d1: 8286285: G1: Rank issues with ParGCRareEvent_lock and Threads_lock
• d478958: 8286179: Node::find(int) should not traverse from new to old nodes
• de8f4d0: 8286191: misc tests fail due to JDK-8285987
• bf0dc4f: 8286367: riscv: riscv port is broken after JDK-8284161
• 4fd79a6: 8285730: unify _WIN32_WINNT settings
• bd6026c: 7124282: [macosx] Can't see table cell highlighter when the highlight border is the same color as the cell.
• ... and 61 more: https://git.openjdk.java.net/jdk/compare/1bb4de2e2868a539846ec48dd43fd623c2ba69a5...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[tstuefe]

I don't understand your "free retained" terminology, but otherwise the changes seem fine and do what you describe.

Thanks, David

Thank you, David.

Free Retained memory is memory you free()'d but which is retained by the glibc, not handed back to the Operating System.

[MBaesken]

Hi Thomas, when looking at the manpage https://man7.org/linux/man-pages/man3/mallinfo.3.html
then the manpage is describing "Total allocated space" as just uordblks , see the output in the example program of the manpage.
So maybe it would be better to print mi.uordblks and mi.hblkhd separately so that someone looking at the mallinfo manpage finds exactly the same terminology described there in the JVM output? Maybe in some situations it is useful too to get both values ?

[tstuefe]

Hi Thomas, when looking at the manpage https://man7.org/linux/man-pages/man3/mallinfo.3.html then the manpage is describing "Total allocated space" as just uordblks , see the output in the example program of the manpage. So maybe it would be better to print mi.uordblks and mi.hblkhd separately so that someone looking at the mallinfo manpage finds exactly the same terminology described there in the JVM output? Maybe in some situations it is useful too to get both values ?

No, it's wrong. Hence this patch (see patch description - the number does not account for large areas allocated by the glibc with mmap). Not sure what the state of that manpage is. But I tested this very thoroughly.

Observe:

1a) Old VM allocates 100mio 16 byte blocks:

thomas@starfish:~$ jjjcmd de.stuefe.repros.AllocCHeap VM.info | grep C-Heap
C-Heap outstanding allocations: 3128240K

OK.

1b) Old VM allocates 16 100mio byte blocks:

thomas@starfish:~$ jjjcmd de.stuefe.repros.AllocCHeap VM.info | grep C-Heap
C-Heap outstanding allocations: 5782K

Not OK. It says 5,7M, but we just allocated (payloadsize) ~1.6G.

2a) Patched VM allocates 100mio 16 byte blocks:

thomas@starfish:~$ jjjcmd de.stuefe.repros.AllocCHeap VM.info | grep C-Heap
C-Heap outstanding allocations: 3171375K, retained: 6440K

Still OK.

2b) Patched VM allocates 16 100mio byte blocks:

thomas@starfish:~$ jjjcmd de.stuefe.repros.AllocCHeap VM.info | grep C-Heap
C-Heap outstanding allocations: 1611294K, retained: 2553K

Now it looks legit too, compared to (1b). The lower size (1.6G vs 3.1G) stems from the massively lower glibc overhead when allocating large areas instead of tiny blocks.

[tstuefe]

Thanks @dholmes-ora and @MBaesken !

/integrate

[openjdk]

Going to push as commit 9e320d9.
Since your change was applied there have been 71 commits pushed to the master branch:

• 65f5067: 8286429: jpackageapplauncher build fails intermittently in Tier[45]
• c4bd449: 8238373: Punctuation should be same in jlink help usage on Japanese language
• 3fa1c40: 8286298: Remove unused methods in sun.invoke.util.VerifyType
• 3462190: 8286163: micro-optimize Instant.plusSeconds
• 60a91d1: 8286285: G1: Rank issues with ParGCRareEvent_lock and Threads_lock
• d478958: 8286179: Node::find(int) should not traverse from new to old nodes
• de8f4d0: 8286191: misc tests fail due to JDK-8285987
• bf0dc4f: 8286367: riscv: riscv port is broken after JDK-8284161
• 4fd79a6: 8285730: unify _WIN32_WINNT settings
• bd6026c: 7124282: [macosx] Can't see table cell highlighter when the highlight border is the same color as the cell.
• ... and 61 more: https://git.openjdk.java.net/jdk/compare/1bb4de2e2868a539846ec48dd43fd623c2ba69a5...master

Your commit was automatically rebased without conflicts.

[openjdk]

@tstuefe Pushed as commit 9e320d9.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.