JDK-8266222: [aix] In mmap-mode, partial releases with os::release_memory may trash internal bookkeeping

[tstuefe]

In os_aix.cpp we keep book about for reserved address ranges since we need to keep information about which API (mmap or shmat) had been used for this range. (code hint: see vmembk_(add|remove|find)).

When releasing memory via os::release_memory, we remove those ranges from the internal bookkeeping.

However, for mmap() the release may be partial. In that case the internal bookkeeping is not updated correctly: we just remove the record for the whole original address range.

After that operation, the remainder address range is still mapped from the point of the VM, but the bookkeeping is lost and subsequent calls to os::release_memory or os::commit/uncommit_memory for this range will run into a guarantee.

Partial releases can happen e.g. when, in the process of running os::reserve_memory_aligned(), the extra-aligned pages are unmapped. This is of more concern now with the new Metaspace, since we now reserve with larger alignments (4G+).

This problem was hidden by the fact that we typically run in 64K paged mode where we use SysV shared memory; but if AME is enabled on AIX, 64K pages are disabled, and the VM falls back to 4K pages and using mmap, and runs into this bug.

Edit: Tested on AIX machines with and without AME activated, solves the problems. All SAP nightlies ran, no problems caused by this patch.

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed

Issue

• JDK-8266222: [aix] In mmap-mode, partial releases with os::release_memory may trash internal bookkeeping

Reviewers

• Martin Doerr (@TheRealMDoerr - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/3765/head:pull/3765
$ git checkout pull/3765

Update a local copy of the PR:
$ git checkout pull/3765
$ git pull https://git.openjdk.java.net/jdk pull/3765/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3765

View PR using the GUI difftool:
$ git pr show -t 3765

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/3765.diff

[bridgekeeper]

👋 Welcome back stuefe! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@tstuefe The following label will be automatically applied to this pull request:

• hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (db9c2695)

[tstuefe]

Reviewer note: this is unrelated to the problem, I just lifted the assert call from both branches and made it independent from debug/release.

[TheRealMDoerr]

This looks good.

[openjdk]

@tstuefe This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8266222: [aix] In mmap-mode, partial releases with os::release_memory may trash internal bookkeeping

Reviewed-by: mdoerr

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 112 new commits pushed to the master branch:

• c9873c4: 8260621: (jrtfs) ThreadLocal memory leak in ImageBufferCache when using jrtfs
• 82768d9: 8221503: vmTestbase/nsk/jdb/eval/eval001/eval001.java fails with: com.sun.jdi.InvalidTypeException: Can't assign double[][][] to double[][][]
• b71f85a: 8264398: BevelBorderUIResource​(int, Color, Color) and BevelBoder(int, Color, Color) spec should clarify about usage of highlight and shadow color
• b172555: 8266171: -Warray-bounds happens in imageioJPEG.c
• 8bcebe7: 8265505: findsym does not work on remote debug server
• b88785d: 8266038: Move newAddress() to JVMDebugger
• 2c53654: 8266179: [macos] jpackage should specify architecture for produced pkg files
• d282799: 8255566: Add size validation when parsing values from VersionProps
• 61365d5: 8266465: Add wildcard to JTwork/JTreport exclude in jib-profiles.js
• f00b70e: 8266527: RandomTestCoverage.java failing due to API removal
• ... and 102 more: https://git.openjdk.java.net/jdk/compare/343a4a76f273078f78897e8fb7e695bc2c111536...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[tstuefe]

Danke, Martin!

[tstuefe]

Note that since we don't have that many AIX reviewers, and this only concerns AIX code, I go ahead and push. Its our port anyway, if something breaks we will have to fix it.

/integrate

[openjdk]

@tstuefe Since your change was applied there have been 114 commits pushed to the master branch:

• 65ce4d2: 8266491: Remove resolve and obj_equals leftovers from BarrierSetAssembler
• a8046c9: 8266436: Synthetic constructor trees have non-null return type
• c9873c4: 8260621: (jrtfs) ThreadLocal memory leak in ImageBufferCache when using jrtfs
• 82768d9: 8221503: vmTestbase/nsk/jdb/eval/eval001/eval001.java fails with: com.sun.jdi.InvalidTypeException: Can't assign double[][][] to double[][][]
• b71f85a: 8264398: BevelBorderUIResource​(int, Color, Color) and BevelBoder(int, Color, Color) spec should clarify about usage of highlight and shadow color
• b172555: 8266171: -Warray-bounds happens in imageioJPEG.c
• 8bcebe7: 8265505: findsym does not work on remote debug server
• b88785d: 8266038: Move newAddress() to JVMDebugger
• 2c53654: 8266179: [macos] jpackage should specify architecture for produced pkg files
• d282799: 8255566: Add size validation when parsing values from VersionProps
• ... and 104 more: https://git.openjdk.java.net/jdk/compare/343a4a76f273078f78897e8fb7e695bc2c111536...master

Your commit was automatically rebased without conflicts.

Pushed as commit 250b45a.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.