Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8303215: Make thread stacks not use huge pages #14105

Closed

Conversation

poonamparhar
Copy link
Member

@poonamparhar poonamparhar commented May 23, 2023

When a system has Transparent Huge Pages (THP) enabled (/sys/kernel/mm/transparent_hugepage/enabled is set to 'always'), thread stacks can have significantly more resident set size (RSS) than they actually require. This occurs when the stack size is 2MB or larger, which makes the memory range of the stack more likely to be aligned on a Large Page Size boundary (2MB on most systems). This in turn makes the stack eligible to be backed by transparent huge pages resulting in more memory consumption than it would otherwise when standard small pages are used. This issue is more apparent on AArch64 platforms where the default stack size is 2MB.

Example mapping from smaps illustrating this issue:
fffced200000-fffced204000 ---p 00000000 00:00 0 
Size:                16 kB     # guard pages
KernelPageSize:       4 kB
MMUPageSize:          4 kB
...
fffced204000-fffced400000 rw-p 00000000 00:00 0 
Size:              2032 kB    # stack space
KernelPageSize:       4 kB
MMUPageSize:          4 kB
Rss:               2032 kB    # entire stack resident in memory

This fix addresses this issue with the following two main changes:

  1. Change the default stack size to 2040KB, which is 2 pages less than 2MB. This ensures that stacks don't get 2MB aligned. And why 2 pages less than 2MB, because for non-JavaThreads, glibc adds an additional guard page to the total stack size. To keep it simple and to keep the default stack size value for all options - ThreadStackSize, CompilerThreadStackSize, and VMThreadStackSize same, we use the default value as 2040K.

Example mapping for a JavaThread:

ffff6e913000-ffff6e917000 ---p 00000000 00:00 0 
Size:                 16 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
...
ffff6e917000-ffff6eb11000 rw-p 00000000 00:00 0 
Size:               2024 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  92 kB

Example Mapping for a non-JavaThread (WatcherThread):

ffff6eb11000-ffff6eb12000 ---p 00000000 00:00 0 
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
...
ffff6eb12000-ffff6ed10000 rw-p 00000000 00:00 0 
Size:               2040 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  12 kB
  1. If the requested stack size is greater than or equal to the default large page size (2MB on most systems) and can be large-page aligned, then add an additional page to the stack size. This reduces the chances of stacks getting large page aligned.

Progress

  • Change must be properly reviewed (1 review required, with at least 1 Reviewer)
  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue

Issue

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/14105/head:pull/14105
$ git checkout pull/14105

Update a local copy of the PR:
$ git checkout pull/14105
$ git pull https://git.openjdk.org/jdk.git pull/14105/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 14105

View PR using the GUI difftool:
$ git pr show -t 14105

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/14105.diff

Webrev

Link to Webrev Comment

@bridgekeeper
Copy link

bridgekeeper bot commented May 23, 2023

👋 Welcome back poonam! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk openjdk bot added the rfr Pull request is ready for review label May 23, 2023
@openjdk
Copy link

openjdk bot commented May 23, 2023

@poonamparhar The following label will be automatically applied to this pull request:

  • hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added the hotspot-runtime hotspot-runtime-dev@openjdk.org label May 23, 2023
@mlbridge
Copy link

mlbridge bot commented May 23, 2023

Webrevs

Copy link
Member

@tstuefe tstuefe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is a pragmatic workaround. I believe @theRealAph recommended the same solution.

Did you test this on an aarch64 kernel with 64k pages? I suspect the 2040 default size only works for 4k or 8k pages, for larger page sizes pthread_create would just round up the stack size again to 2MB.

src/hotspot/os/linux/os_linux.cpp Show resolved Hide resolved
@poonamparhar
Copy link
Member Author

Yes, this is a pragmatic workaround. I believe @theRealAph recommended the same solution.

Did you test this on an aarch64 kernel with 64k pages? I suspect the 2040 default size only works for 4k or 8k pages, for larger page sizes pthread_create would just round up the stack size again to 2MB.

This does not solve the problem when the default small page size is 64K. The stack size gets rounded up to 2048K.

JavaThread:

0000ffff18ff0000     192       0       0 -----   [ anon ]
0000ffff19020000    1856     192     192 rw---   [ anon ]
0000ffff191f0000     192       0       0 -----   [ anon ]
0000ffff19220000    1856     192     192 rw---   [ anon ]
0000ffff193f0000     192       0       0 -----   [ anon ]
0000ffff19420000    1856     192     192 rw---   [ anon ]

non-JavaThread:

0000ffff6c170000      64       0       0 -----   [ anon ]
0000ffff6c180000    2048      64      64 rw---   [ anon ]
0000ffff6c380000      64       0       0 -----   [ anon ]
0000ffff6c390000    2048      64      64 rw---   [ anon ]

For 64k page size, changing the stack size to 1920K (2048 - 2*64) would work:

0000fffdc5ba0000     192       0       0 -----   [ anon ]
0000fffdc5bd0000    1728     192     192 rw---   [ anon ]
0000fffdc5d80000     192       0       0 -----   [ anon ]
0000fffdc5db0000    1728     192     192 rw---   [ anon ]
0000fffdc5f60000     192       0       0 -----   [ anon ]
0000fffdc5f90000    1728     192     192 rw---   [ anon ]

@poonamparhar
Copy link
Member Author

poonamparhar commented May 24, 2023

For the 64K page size, the default huge page size is usually 512M. The aarch64 machine with 64K page size that I am testing on has:

bash-4.4$ cat /proc/meminfo | grep Hugepagesize
Hugepagesize:     524288 kB

And on that, the current set of changes behave correctly when I run with -Xss512m. Snippet of pmap output:

0000fff8b7ff0000     192       0       0 -----   [ anon ]  # guard pages
0000fff8b8020000  524160     192     192 rw---   [ anon ]  # usable stack area
0000fff8d8000000   65472   65472   65472 rw---   [ anon ]
0000fff8dbff0000      64       0       0 -----   [ anon ]
0000fff8dfff0000     192       0       0 -----   [ anon ]
0000fff8e0020000  524160     192     192 rw---   [ anon ]

an additional 64k page gets added to the stack size to disturb the large page alignment.

@iklam
Copy link
Member

iklam commented May 25, 2023

Does this PR affect only Linux/aarch64? If so, I think it's better to add that into the bug title.

@tstuefe
Copy link
Member

tstuefe commented May 25, 2023

Does this PR affect only Linux/aarch64? If so, I think it's better to add that into the bug title.

It affects all platforms, reproducible if you set stack size to 2M and THP to always. Aarch64 has 2M as default though.

Copy link
Member

@tstuefe tstuefe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some small nits remain.

src/hotspot/os/linux/os_linux.cpp Outdated Show resolved Hide resolved
src/hotspot/os/linux/os_linux.cpp Outdated Show resolved Hide resolved
Copy link
Member

@tstuefe tstuefe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks for fixing.

@openjdk
Copy link

openjdk bot commented May 27, 2023

@poonamparhar This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8303215: Make thread stacks not use huge pages

Reviewed-by: stuefe, dholmes

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 122 new commits pushed to the master branch:

  • 0119969: 8309171: Test vmTestbase/nsk/jvmti/scenarios/jni_interception/JI05/ji05t001/TestDescription.java fails after JDK-8308341
  • f8a924a: 8308975: Fix signed integer overflow in compiler code, part 2
  • 5531f6b: 8308819: add JDWP and JDI virtual thread support for ThreadReference.ForceEarlyReturn
  • e42a4b6: 8309236: ProblemList java/util/concurrent/locks/Lock/OOMEInAQS.java with ZGC and Generational ZGC again
  • 8dbd384: 8308678: (fs) UnixPath::toRealPath needs additional permissions when running with SM (macOS)
  • c3cd481: 8304914: Use OperatingSystem, Architecture, and Version in jpackage
  • eae1f59: 8309159: Some minor comment and code cleanup in jdk/com/sun/jdi/PopFramesTest.java
  • 45473ef: 8309230: ProblemList jdk/incubator/vector/Float64VectorTests.java on aarch64
  • 78aa5f3: 8299505: findVirtual on array classes incorrectly restricts the receiver type
  • 42ca6e6: 8308022: update for deprecated sprintf for java.base
  • ... and 112 more: https://git.openjdk.org/jdk/compare/bddf48380e658df630fecad5eda40106a24b6e1c...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready Pull request is ready to be integrated label May 27, 2023
Comment on lines 3757 to 3759

// 1) Handle the case where we do not want to use huge pages and hence
// there is no need to scan the OS for related info
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment needs updating as now you do the scan first.

Comment on lines 932 to 934
if (stack_size >= os::Linux::default_large_page_size() && is_aligned(stack_size, os::Linux::default_large_page_size())) {
stack_size += os::vm_page_size();
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to watch for a zero default size here so that we don't expand all stacks by a page?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a check for zero default page size. Please review.

@@ -3745,6 +3751,10 @@ bool os::Linux::setup_large_page_type(size_t page_size) {
}

void os::large_page_init() {
// Scan default large page size
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion:

// Always initialize the default large page size even if large pages are not being used.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the comment.


define_pd_global(intx, CompilerThreadStackSize, 2048);
// set default stack sizes < 2MB so as to prevent stacks from getting
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/set/Set/

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

define_pd_global(intx, CompilerThreadStackSize, 2048);
// set default stack sizes < 2MB so as to prevent stacks from getting
// large-page aligned and backed by THPs on systems where 2MB is the
// default huge page size. For non-JavaThreads, glibc adds an additional
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/adds/may add/

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

src/hotspot/os_cpu/linux_aarch64/globals_linux_aarch64.hpp Outdated Show resolved Hide resolved
Copy link
Member

@dholmes-ora dholmes-ora left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor nit but otherwise nothing further from me.

Thanks.

src/hotspot/os/linux/os_linux.cpp Outdated Show resolved Hide resolved
@poonamparhar
Copy link
Member Author

@tstuefe Hi Thomas, I have addressed the review comments from David Holmes. Could you please take a look at the updated changes again? Thanks!

@tstuefe
Copy link
Member

tstuefe commented Jun 2, 2023

@tstuefe Hi Thomas, I have addressed the review comments from David Holmes. Could you please take a look at the updated changes again? Thanks!

Still good. Thanks.

@poonamparhar
Copy link
Member Author

/integrate

@openjdk
Copy link

openjdk bot commented Jun 2, 2023

Going to push as commit 59d9d9f.
Since your change was applied there have been 155 commits pushed to the master branch:

  • cb1e5e3: 8309286: G1: Remove unused G1HeapRegionAttr::is_valid_gen
  • e8268d9: 8309210: Extend VM Operations hs_err logging
  • 7dbdad5: 8308892: Bad graph detected in build_loop_late after JDK-8305635
  • dc8bc6c: 8308090: Add container tests for on-the-fly resource quota updates
  • 73e7af9: 8309287: Add fontconfig requirement to building.md for Debian
  • aeb53e6: 8308711: Develop additional Tests for KEM implementation
  • dcd9590: 8309224: Fix xlc17 clang 15 warnings in java.desktop
  • 8f1ce78: 8308752: Generational ZGC: Avoid final marking through stack chunks
  • 7b0a336: 8308387: CLD created and unloading list sharing _next node pointer leads to concurrent YC missing CLD roots
  • 60f3b87: 8309295: C2: MaxNode::signed_min() returns nullptr for int operands
  • ... and 145 more: https://git.openjdk.org/jdk/compare/bddf48380e658df630fecad5eda40106a24b6e1c...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot added the integrated Pull request has been integrated label Jun 2, 2023
@openjdk openjdk bot closed this Jun 2, 2023
@openjdk openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Jun 2, 2023
@openjdk
Copy link

openjdk bot commented Jun 2, 2023

@poonamparhar Pushed as commit 59d9d9f.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hotspot-runtime hotspot-runtime-dev@openjdk.org integrated Pull request has been integrated
4 participants