8303215: Make thread stacks not use huge pages #14105

poonamparhar · 2023-05-23T18:01:51Z

When a system has Transparent Huge Pages (THP) enabled (/sys/kernel/mm/transparent_hugepage/enabled is set to 'always'), thread stacks can have significantly more resident set size (RSS) than they actually require. This occurs when the stack size is 2MB or larger, which makes the memory range of the stack more likely to be aligned on a Large Page Size boundary (2MB on most systems). This in turn makes the stack eligible to be backed by transparent huge pages resulting in more memory consumption than it would otherwise when standard small pages are used. This issue is more apparent on AArch64 platforms where the default stack size is 2MB.

Example mapping from smaps illustrating this issue:
fffced200000-fffced204000 ---p 00000000 00:00 0 
Size:                16 kB     # guard pages
KernelPageSize:       4 kB
MMUPageSize:          4 kB
...
fffced204000-fffced400000 rw-p 00000000 00:00 0 
Size:              2032 kB    # stack space
KernelPageSize:       4 kB
MMUPageSize:          4 kB
Rss:               2032 kB    # entire stack resident in memory

This fix addresses this issue with the following two main changes:

Change the default stack size to 2040KB, which is 2 pages less than 2MB. This ensures that stacks don't get 2MB aligned. And why 2 pages less than 2MB, because for non-JavaThreads, glibc adds an additional guard page to the total stack size. To keep it simple and to keep the default stack size value for all options - ThreadStackSize, CompilerThreadStackSize, and VMThreadStackSize same, we use the default value as 2040K.

Example mapping for a JavaThread:

ffff6e913000-ffff6e917000 ---p 00000000 00:00 0 
Size:                 16 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
...
ffff6e917000-ffff6eb11000 rw-p 00000000 00:00 0 
Size:               2024 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  92 kB

Example Mapping for a non-JavaThread (WatcherThread):

ffff6eb11000-ffff6eb12000 ---p 00000000 00:00 0 
Size:                  4 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
...
ffff6eb12000-ffff6ed10000 rw-p 00000000 00:00 0 
Size:               2040 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  12 kB

If the requested stack size is greater than or equal to the default large page size (2MB on most systems) and can be large-page aligned, then add an additional page to the stack size. This reduces the chances of stacks getting large page aligned.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8303215: Make thread stacks not use huge pages

Reviewers

Thomas Stuefe (@tstuefe - Reviewer) ⚠️ Review applies to a4698247
David Holmes (@dholmes-ora - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/14105/head:pull/14105
$ git checkout pull/14105

Update a local copy of the PR:
$ git checkout pull/14105
$ git pull https://git.openjdk.org/jdk.git pull/14105/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 14105

View PR using the GUI difftool:
$ git pr show -t 14105

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/14105.diff

Webrev

Link to Webrev Comment

bridgekeeper · 2023-05-23T18:03:12Z

👋 Welcome back poonam! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2023-05-23T18:06:27Z

@poonamparhar The following label will be automatically applied to this pull request:

hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2023-05-23T18:12:29Z

Webrevs

tstuefe

Yes, this is a pragmatic workaround. I believe @theRealAph recommended the same solution.

Did you test this on an aarch64 kernel with 64k pages? I suspect the 2040 default size only works for 4k or 8k pages, for larger page sizes pthread_create would just round up the stack size again to 2MB.

src/hotspot/os/linux/os_linux.cpp

poonamparhar · 2023-05-23T18:42:33Z

Yes, this is a pragmatic workaround. I believe @theRealAph recommended the same solution.

Did you test this on an aarch64 kernel with 64k pages? I suspect the 2040 default size only works for 4k or 8k pages, for larger page sizes pthread_create would just round up the stack size again to 2MB.

This does not solve the problem when the default small page size is 64K. The stack size gets rounded up to 2048K.

JavaThread:

0000ffff18ff0000     192       0       0 -----   [ anon ]
0000ffff19020000    1856     192     192 rw---   [ anon ]
0000ffff191f0000     192       0       0 -----   [ anon ]
0000ffff19220000    1856     192     192 rw---   [ anon ]
0000ffff193f0000     192       0       0 -----   [ anon ]
0000ffff19420000    1856     192     192 rw---   [ anon ]

non-JavaThread:

0000ffff6c170000      64       0       0 -----   [ anon ]
0000ffff6c180000    2048      64      64 rw---   [ anon ]
0000ffff6c380000      64       0       0 -----   [ anon ]
0000ffff6c390000    2048      64      64 rw---   [ anon ]

For 64k page size, changing the stack size to 1920K (2048 - 2*64) would work:

0000fffdc5ba0000     192       0       0 -----   [ anon ]
0000fffdc5bd0000    1728     192     192 rw---   [ anon ]
0000fffdc5d80000     192       0       0 -----   [ anon ]
0000fffdc5db0000    1728     192     192 rw---   [ anon ]
0000fffdc5f60000     192       0       0 -----   [ anon ]
0000fffdc5f90000    1728     192     192 rw---   [ anon ]

poonamparhar · 2023-05-24T18:47:25Z

For the 64K page size, the default huge page size is usually 512M. The aarch64 machine with 64K page size that I am testing on has:

bash-4.4$ cat /proc/meminfo | grep Hugepagesize
Hugepagesize:     524288 kB

And on that, the current set of changes behave correctly when I run with -Xss512m. Snippet of pmap output:

0000fff8b7ff0000     192       0       0 -----   [ anon ]  # guard pages
0000fff8b8020000  524160     192     192 rw---   [ anon ]  # usable stack area
0000fff8d8000000   65472   65472   65472 rw---   [ anon ]
0000fff8dbff0000      64       0       0 -----   [ anon ]
0000fff8dfff0000     192       0       0 -----   [ anon ]
0000fff8e0020000  524160     192     192 rw---   [ anon ]

an additional 64k page gets added to the stack size to disturb the large page alignment.

iklam · 2023-05-25T06:11:02Z

Does this PR affect only Linux/aarch64? If so, I think it's better to add that into the bug title.

tstuefe · 2023-05-25T08:50:31Z

Does this PR affect only Linux/aarch64? If so, I think it's better to add that into the bug title.

It affects all platforms, reproducible if you set stack size to 2M and THP to always. Aarch64 has 2M as default though.

…it()

tstuefe

Some small nits remain.

src/hotspot/os/linux/os_linux.cpp

src/hotspot/os_cpu/linux_aarch64/globals_linux_aarch64.hpp

tstuefe

Looks good. Thanks for fixing.

openjdk · 2023-05-27T05:36:36Z

@poonamparhar This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8303215: Make thread stacks not use huge pages

Reviewed-by: stuefe, dholmes

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 122 new commits pushed to the master branch:

0119969: 8309171: Test vmTestbase/nsk/jvmti/scenarios/jni_interception/JI05/ji05t001/TestDescription.java fails after JDK-8308341
f8a924a: 8308975: Fix signed integer overflow in compiler code, part 2
5531f6b: 8308819: add JDWP and JDI virtual thread support for ThreadReference.ForceEarlyReturn
e42a4b6: 8309236: ProblemList java/util/concurrent/locks/Lock/OOMEInAQS.java with ZGC and Generational ZGC again
8dbd384: 8308678: (fs) UnixPath::toRealPath needs additional permissions when running with SM (macOS)
c3cd481: 8304914: Use OperatingSystem, Architecture, and Version in jpackage
eae1f59: 8309159: Some minor comment and code cleanup in jdk/com/sun/jdi/PopFramesTest.java
45473ef: 8309230: ProblemList jdk/incubator/vector/Float64VectorTests.java on aarch64
78aa5f3: 8299505: findVirtual on array classes incorrectly restricts the receiver type
42ca6e6: 8308022: update for deprecated sprintf for java.base
... and 112 more: https://git.openjdk.org/jdk/compare/bddf48380e658df630fecad5eda40106a24b6e1c...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

dholmes-ora · 2023-05-29T01:16:00Z

src/hotspot/os/linux/os_linux.cpp

+
  // 1) Handle the case where we do not want to use huge pages and hence
  //    there is no need to scan the OS for related info


The comment needs updating as now you do the scan first.

dholmes-ora · 2023-05-29T01:19:57Z

src/hotspot/os/linux/os_linux.cpp

+  if (stack_size >= os::Linux::default_large_page_size() && is_aligned(stack_size, os::Linux::default_large_page_size())) {
+    stack_size += os::vm_page_size();
+  }


Do we need to watch for a zero default size here so that we don't expand all stacks by a page?

Added a check for zero default page size. Please review.

dholmes-ora · 2023-05-29T01:20:19Z

src/hotspot/os/linux/os_linux.cpp

 }

 void os::large_page_init() {
+  // Scan default large page size


Suggestion:

// Always initialize the default large page size even if large pages are not being used.

Changed the comment.

dholmes-ora · 2023-05-29T01:35:10Z

src/hotspot/os_cpu/linux_aarch64/globals_linux_aarch64.hpp

-define_pd_global(intx, VMThreadStackSize,        2048);

-define_pd_global(intx, CompilerThreadStackSize,  2048);
+// set default stack sizes < 2MB so as to prevent stacks from getting


dholmes-ora · 2023-05-29T01:35:30Z

src/hotspot/os_cpu/linux_aarch64/globals_linux_aarch64.hpp

-define_pd_global(intx, CompilerThreadStackSize,  2048);
+// set default stack sizes < 2MB so as to prevent stacks from getting
+// large-page aligned and backed by THPs on systems where 2MB is the
+// default huge page size. For non-JavaThreads, glibc adds an additional


s/adds/may add/

src/hotspot/os_cpu/linux_aarch64/globals_linux_aarch64.hpp

dholmes-ora

One minor nit but otherwise nothing further from me.

Thanks.

src/hotspot/os/linux/os_linux.cpp

poonamparhar · 2023-05-31T22:04:28Z

@tstuefe Hi Thomas, I have addressed the review comments from David Holmes. Could you please take a look at the updated changes again? Thanks!

tstuefe · 2023-06-02T11:36:07Z

@tstuefe Hi Thomas, I have addressed the review comments from David Holmes. Could you please take a look at the updated changes again? Thanks!

Still good. Thanks.

poonamparhar · 2023-06-02T13:31:25Z

/integrate

openjdk · 2023-06-02T13:32:29Z

Going to push as commit 59d9d9f.
Since your change was applied there have been 155 commits pushed to the master branch:

cb1e5e3: 8309286: G1: Remove unused G1HeapRegionAttr::is_valid_gen
e8268d9: 8309210: Extend VM Operations hs_err logging
7dbdad5: 8308892: Bad graph detected in build_loop_late after JDK-8305635
dc8bc6c: 8308090: Add container tests for on-the-fly resource quota updates
73e7af9: 8309287: Add fontconfig requirement to building.md for Debian
aeb53e6: 8308711: Develop additional Tests for KEM implementation
dcd9590: 8309224: Fix xlc17 clang 15 warnings in java.desktop
8f1ce78: 8308752: Generational ZGC: Avoid final marking through stack chunks
7b0a336: 8308387: CLD created and unloading list sharing _next node pointer leads to concurrent YC missing CLD roots
60f3b87: 8309295: C2: MaxNode::signed_min() returns nullptr for int operands
... and 145 more: https://git.openjdk.org/jdk/compare/bddf48380e658df630fecad5eda40106a24b6e1c...master

Your commit was automatically rebased without conflicts.

openjdk · 2023-06-02T13:32:40Z

@poonamparhar Pushed as commit 59d9d9f.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8303215: Make thread stacks not use huge pages

63fa74a

openjdk bot added the rfr Pull request is ready for review label May 23, 2023

openjdk bot added the hotspot-runtime hotspot-runtime-dev@openjdk.org label May 23, 2023

tstuefe reviewed May 23, 2023

View reviewed changes

src/hotspot/os/linux/os_linux.cpp Show resolved Hide resolved

call scan_default_large_page_size() in the beginning of large_page_in…

c1353af

…it()

tstuefe suggested changes May 25, 2023

View reviewed changes

src/hotspot/os/linux/os_linux.cpp Outdated Show resolved Hide resolved

src/hotspot/os/linux/os_linux.cpp Outdated Show resolved Hide resolved

src/hotspot/os_cpu/linux_aarch64/globals_linux_aarch64.hpp Show resolved Hide resolved

addressed review comments

a469824

tstuefe approved these changes May 27, 2023

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label May 27, 2023

dholmes-ora suggested changes May 29, 2023

View reviewed changes

addressed second set of review comments

7643e8d

dholmes-ora approved these changes May 31, 2023

View reviewed changes

src/hotspot/os/linux/os_linux.cpp Outdated Show resolved Hide resolved

combined the if-clause statements

1ef7143

dholmes-ora approved these changes May 31, 2023

View reviewed changes

openjdk bot added the integrated Pull request has been integrated label Jun 2, 2023

openjdk bot closed this Jun 2, 2023

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Jun 2, 2023


		// 1) Handle the case where we do not want to use huge pages and hence
		// there is no need to scan the OS for related info

8303215: Make thread stacks not use huge pages #14105

8303215: Make thread stacks not use huge pages #14105

Uh oh!

Conversation

poonamparhar commented May 23, 2023 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Reviewing

Webrev

Uh oh!

bridgekeeper bot commented May 23, 2023

Uh oh!

openjdk bot commented May 23, 2023

Uh oh!

mlbridge bot commented May 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

tstuefe left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

poonamparhar commented May 23, 2023

Uh oh!

poonamparhar commented May 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iklam commented May 25, 2023

Uh oh!

tstuefe commented May 25, 2023

Uh oh!

tstuefe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tstuefe left a comment

Choose a reason for hiding this comment

Uh oh!

openjdk bot commented May 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dholmes-ora left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

poonamparhar commented May 31, 2023

Uh oh!

tstuefe commented Jun 2, 2023

Uh oh!

poonamparhar commented Jun 2, 2023

Uh oh!

openjdk bot commented Jun 2, 2023

Uh oh!

openjdk bot commented Jun 2, 2023

Uh oh!

Reviewers

poonamparhar commented May 23, 2023 •

edited by openjdk bot

Loading

mlbridge bot commented May 23, 2023 •

edited

Loading

tstuefe left a comment •

edited

Loading

poonamparhar commented May 24, 2023 •

edited

Loading

openjdk bot commented May 27, 2023 •

edited

Loading