8364518: Support for Job Objects in os::commit_memory_limit() on Windows

[jsikstro]

Hello,

We currently use os::commit_memory_limit() to limit the initial, min and max values when ergonomically setting the heap size. Right now, os::commit_memory_limit() returns the available virtual address space on Windows, which is guaranteed to be the upper-bound of how much memory can be committed. However, this does not take into account user-configurable limits.

I propose revising os::commit_memory_limit() to consider Windows Job Objects. Job Objects can, among other things, restrict the amount of memory that can be committed by a single process or all processes in a job. If we are unable to query Job Object information or if no limit is set, I suggest returning SIZE_MAX, since the next effective limit is the available virtual address space. The size of this virtual address space cannot be modified by the user, except by consuming it.

For more details and short examples, I have created a GitHub repository summarizing my findings regarding memory limits and Job Objects on Windows: https://github.com/jsikstro/MemoryLimitsWindows .

Adding support for Job Objects in this way adds support for detecting memory limits when using native Windows Hyper-V containers, which uses Job Objects to limit memory.

Testing:

• Oracle's tier1-2
• Manual testing on Windows using both Process Governor (a popular tool), and creating a Job Object manually.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8364518: Support for Job Objects in os::commit_memory_limit() on Windows (Enhancement - P4)

Reviewers

• Albert Mingkun Yang (@albertnetymk - Reviewer)
• David Holmes (@dholmes-ora - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/26593/head:pull/26593
$ git checkout pull/26593

Update a local copy of the PR:
$ git checkout pull/26593
$ git pull https://git.openjdk.org/jdk.git pull/26593/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 26593

View PR using the GUI difftool:
$ git pr show -t 26593

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/26593.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back jsikstro! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@jsikstro This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8364518: Support for Job Objects in os::commit_memory_limit() on Windows

Reviewed-by: ayang, dholmes

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 53 new commits pushed to the master branch:

• ca41644: 8355379: Annotate lazy fields in java.security @stable
• 6656e76: 8359820: Improve handshake/safepoint timeout diagnostic messages
• 68a3551: 8364212: Shenandoah: Rework archived objects loading
• ... and 50 more: https://git.openjdk.org/jdk/compare/ae11d8f44689502d35cb511e9ce288ab7cc0acae...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[openjdk]

@jsikstro The following label will be automatically applied to this pull request:

• hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (eb446664)
• 00: Full (436e2160)

[jsikstro]

/cc hotspot-gc

[openjdk]

@jsikstro
The hotspot-gc label was successfully added.

[dholmes-ora]

I can't help but think that this should be part of a broader effort to support "containers" on Windows. We allowed the use of JobObjects for the processor count because it was similar what you could do with cpu-sets on Linux and not fundamentally "container" related. What you propose here looks good, but surely there are more things that need to be adjusted to account for JobObjects in this space? Case in point I just raised a query today about why we have a confused notion of "physical memory" on Linux, depending on whether you are running in a container or not. I would suspect a similar distinction would need to be made on Windows if running under JobObjects.

To be clear I have no issue with the current PR providing this part of Windows "container" support, but I would like to see at least an Umbrella JBS issue (JEP?) created to cover everything that would be needed for such support. Thanks

[jsikstro]

Thank you for your input @dholmes-ora, and I agree that broader support for Job Objects, and by extension, native Windows containers, are possible in this space. I have nothing against creating an umbrella issue that covers adding support for native Windows containers. However, such containers were not my main focus, but an added benefit I suppose.

Using the size of the available virtual address space is a good indicator on how much memory can be committed, and will always be the absolute upper-bound. However, reading limits set by Job Object(s) is more complete IMO, since it (is the only way to?) limit how much memory can be committed, which is what this function is aiming to answer. So even though I think this overlaps with native Windows container support, I think this is a standalone enhancement as well.

[dholmes-ora]

This looks good to me. I have filed

JDK-8364803 Provide support for Windows Job Objects that limit process resources

as the umbrella task for Job Object support.

Thanks

[jsikstro]

Thank you for the reviews! @albertnetymk @dholmes-ora
/integrate

[openjdk]

Going to push as commit 8d529bc.
Since your change was applied there have been 53 commits pushed to the master branch:

• ca41644: 8355379: Annotate lazy fields in java.security @stable
• 6656e76: 8359820: Improve handshake/safepoint timeout diagnostic messages
• 68a3551: 8364212: Shenandoah: Rework archived objects loading
• ... and 50 more: https://git.openjdk.org/jdk/compare/ae11d8f44689502d35cb511e9ce288ab7cc0acae...master

Your commit was automatically rebased without conflicts.

[openjdk]

@jsikstro Pushed as commit 8d529bc.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.