8310110: Shenandoah: Trace page sizes #14486

tstuefe · 2023-06-15T07:37:11Z

-Xlog:pagesize now produced this printout

[0.006s][info][pagesize] Heap: req_size=5G base=0x00000006c0000000 page_size=1G alignment=1G size=5G
[0.006s][info][pagesize] Mark Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
[0.011s][info][pagesize] Aux Bitmap: req_size=160M base=0x00007fc4c0000000 page_size=1G alignment=1G size=1G
[0.011s][info][pagesize] Region Storage: req_size=320K base=0x00007fc3c0000000 page_size=1G alignment=1G size=1G
[0.011s][info][pagesize] Collection Set: req_size=16K base=0x0000000000010000 page_size=4K alignment=4K size=16K

Note that this printout is from a machine with 1G pages and -XX:+UseLargePages; it nicely demonstrates how much memory we waste in that case on support structures; https://bugs.openjdk.org/browse/JDK-8310111 tracks that.

Also note that I reuse os::trace_page_sizes_for_requested_size here, which I think can be improved and made clearer (alignment is actually requested pagesize). This is tracked separately in #14484 .

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8310110: Shenandoah: Trace page sizes (Enhancement - P4)

Reviewers

Aleksey Shipilev (@shipilev - Reviewer)
Roman Kennke (@rkennke - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/14486/head:pull/14486
$ git checkout pull/14486

Update a local copy of the PR:
$ git checkout pull/14486
$ git pull https://git.openjdk.org/jdk.git pull/14486/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 14486

View PR using the GUI difftool:
$ git pr show -t 14486

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/14486.diff

Webrev

Link to Webrev Comment

bridgekeeper · 2023-06-15T07:38:22Z

👋 Welcome back stuefe! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2023-06-15T07:39:49Z

@tstuefe The following labels will be automatically applied to this pull request:

hotspot
shenandoah

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

mlbridge · 2023-06-16T07:07:59Z

Webrevs

shipilev

The idea looks good.

I would prefer not to have a one-liner helper method, and instead inline the uses like this:

   if (ShenandoahVerify) {
     ReservedSpace verify_bitmap(_bitmap_size, bitmap_page_size);
+    os::trace_page_sizes_for_requested_size("Verify Bitmap", bitmap_size_orig,
+                                            bitmap.page_size(), bitmap_page_size,
+                                            bitmap.base(), bitmap.size());
     if (!verify_bitmap.special()) {

Also, let's wait for #14484 to land first?

shipilev · 2023-06-16T09:49:01Z

Oh, also, we try to keep the synopsis format like this: "8310110: Shenandoah: Trace page sizes"

tstuefe · 2023-06-16T14:18:34Z

The idea looks good.

I would prefer not to have a one-liner helper method, and instead inline the uses like this:

   if (ShenandoahVerify) {
     ReservedSpace verify_bitmap(_bitmap_size, bitmap_page_size);
+    os::trace_page_sizes_for_requested_size("Verify Bitmap", bitmap_size_orig,
+                                            bitmap.page_size(), bitmap_page_size,
+                                            bitmap.base(), bitmap.size());
     if (!verify_bitmap.special()) {

Okay.

Also, let's wait for #14484 to land first?

Certainly. Mind reviewing #14484 ? It's very simple.

tstuefe · 2023-06-19T07:11:05Z

@shipilev Thanks for reviewing. Here you go.

shipilev

Looks okay to me!

I would prefer the style to be:

  os::trace_page_sizes_for_requested_size("Mark Bitmap",
                                          bitmap_size_orig, bitmap_page_size,
                                          bitmap.base(),
                                          bitmap.size(), bitmap.page_size());

...so that we have:

line 1: label
line 2: original size, page size
line 3: base addr
line 4: actual size, page size

Would be easier to check lines 2 and 4 match?

openjdk · 2023-06-19T08:08:35Z

@tstuefe This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8310110: Shenandoah: Trace page sizes

Reviewed-by: shade, rkennke

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 109 new commits pushed to the master branch:

c3f10e8: 8307625: Redundant receiver null check in LibraryCallKit::generate_method_call
39c104d: 8310380: Handle problems in core-related tests on macOS when codesign tool does not work
526dba1: 8310130: C2: assert(false) failed: scalar_input is neither phi nor a matchin reduction
48e61c1: 8310728: Enable Zc:inline flag in Visual Studio build
56a73a6: 8309591: Socket.setOption(TCP_QUICKACK) uses wrong level
afdaa2a: 8309109: AArch64: [TESTBUG] compiler/intrinsics/sha/cli/TestUseSHA3IntrinsicsOptionOnSupportedCPU.java fails on Neoverse N2 and V1
7f09435: 8310982: jdk/internal/util/ArchTest.java fails after JDK-8308452 failed with Method isARM()
f07e396: 8310914: Remove 2 malformed java/foreign ProblemList entries
a97f98f: 8310585: GetThreadState spec mentions undefined JVMTI_THREAD_STATE_MONITOR_WAITING
2cf5c2f: 8310238: [test bug] javax/swing/JTableHeader/6889007/bug6889007.java fails
... and 99 more: https://git.openjdk.org/jdk/compare/4a9cc8a000cafb3ad77a33710054b567e8553652...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

tstuefe · 2023-06-19T08:26:10Z

Sure, its your code :-) I reshaped the callsites.

shipilev

Looks good, thanks!

Please make sure Windows builds/tests pass -- I think GHA in current master should be good. I see you merged recently, so we might just wait for GHA to complete.

tstuefe · 2023-06-20T14:09:46Z

The newly added Shenandoah test uncovered an old bug where the Aux bitmap would not be madvised correctly for use with THP. Tracked by https://bugs.openjdk.org/browse/JDK-8310388, and I disable the test for now.

shipilev · 2023-06-20T14:40:43Z

The newly added Shenandoah test uncovered an old bug where the Aux bitmap would not be madvised correctly for use with THP. Tracked by https://bugs.openjdk.org/browse/JDK-8310388, and I disable the test for now.

OK, assuming we get the consensus around JDK-8310388 soon, we can wait for that to land before this test? :)

tstuefe · 2023-06-20T15:02:03Z

The newly added Shenandoah test uncovered an old bug where the Aux bitmap would not be madvised correctly for use with THP. Tracked by https://bugs.openjdk.org/browse/JDK-8310388, and I disable the test for now.

OK, assuming we get the consensus around JDK-8310388 soon, we can wait for that to land before this test? :)

The test still makes sense for explicit huge pages, or?

shipilev · 2023-06-20T15:04:02Z

The test still makes sense for explicit huge pages, or?

Yes, it does. I just want to minimize churn and simplify backports :) So fewer issues to backport with fewer dependencies would be better.

tstuefe · 2023-06-20T15:10:32Z

The test still makes sense for explicit huge pages, or?

Yes, it does. I just want to minimize churn and simplify backports :) So fewer issues to backport with fewer dependencies would be better.

I'll remove the test from this PR.

shipilev

All right, this works too.

tstuefe · 2023-06-27T15:15:35Z

Ping... may I have a second review? This is simple stuff.

rkennke · 2023-06-28T09:05:31Z

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp

      if (cset_rs.is_reserved()) {
        assert(cset_rs.base() == req_addr, "Allocated where requested: " PTR_FORMAT ", " PTR_FORMAT, p2i(cset_rs.base()), addr);
        _collection_set = new ShenandoahCollectionSet(this, cset_rs, sh_rs.base());
+        rs = &cset_rs;


I'm probably missing something here, but doesn't cset_rs go out of scope, and rs would point to undefined stuff?

:-( you are of course right. This only worked out of accident since ReservedSpace has no dtor.

:-( you are of course right. This only worked out of accident since ReservedSpace has no dtor.

Could you copy-assign the object, instead?

Or just null-check the resulting _collection_set and print the message then. Don't need to mess with locals then?

Okay, fixed. Copying ReservedSpace is fine, we do it in other places too. This thing is used like a value object in most places.

rkennke

Looks good now, thanks!

tstuefe · 2023-06-28T11:56:07Z

I verified manually that macos m1 builds.

Thanks @shipilev and @rkennke!

/integrate

openjdk · 2023-06-28T11:57:22Z

Going to push as commit ef71c32.
Since your change was applied there have been 109 commits pushed to the master branch:

c3f10e8: 8307625: Redundant receiver null check in LibraryCallKit::generate_method_call
39c104d: 8310380: Handle problems in core-related tests on macOS when codesign tool does not work
526dba1: 8310130: C2: assert(false) failed: scalar_input is neither phi nor a matchin reduction
48e61c1: 8310728: Enable Zc:inline flag in Visual Studio build
56a73a6: 8309591: Socket.setOption(TCP_QUICKACK) uses wrong level
afdaa2a: 8309109: AArch64: [TESTBUG] compiler/intrinsics/sha/cli/TestUseSHA3IntrinsicsOptionOnSupportedCPU.java fails on Neoverse N2 and V1
7f09435: 8310982: jdk/internal/util/ArchTest.java fails after JDK-8308452 failed with Method isARM()
f07e396: 8310914: Remove 2 malformed java/foreign ProblemList entries
a97f98f: 8310585: GetThreadState spec mentions undefined JVMTI_THREAD_STATE_MONITOR_WAITING
2cf5c2f: 8310238: [test bug] javax/swing/JTableHeader/6889007/bug6889007.java fails
... and 99 more: https://git.openjdk.org/jdk/compare/4a9cc8a000cafb3ad77a33710054b567e8553652...master

Your commit was automatically rebased without conflicts.

openjdk · 2023-06-28T11:57:28Z

@tstuefe Pushed as commit ef71c32.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

openjdk bot added hotspot hotspot-dev@openjdk.org shenandoah shenandoah-dev@openjdk.org labels Jun 15, 2023

JDK-8310110-Trace-page-sizes

fd7ecbc

tstuefe force-pushed the JDK-8310110-Trace-page-sizes branch from 3c6e5f1 to fd7ecbc Compare June 16, 2023 06:49

tstuefe marked this pull request as ready for review June 16, 2023 07:01

openjdk bot added the rfr Pull request is ready for review label Jun 16, 2023

shipilev reviewed Jun 16, 2023

View reviewed changes

tstuefe changed the title ~~JDK-8310110: ShenandoahGC: Trace page sizes~~ 8310110: Shenandoah: Trace page sizes Jun 16, 2023

tstuefe added 3 commits June 19, 2023 08:05

Merge branch 'master' into JDK-8310110-Trace-page-sizes

afca584

feedback aleksey

e2a776d

enable runtime/os/TestTracePageSizes.java for shenandoah

493ba7e

shipilev approved these changes Jun 19, 2023

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Jun 19, 2023

reformat

2a4df08

shipilev approved these changes Jun 19, 2023

View reviewed changes

disable Shenandoah THP trace test for now

07e113f

remove Shenandoah from TestTracePageSizes

699eb26

tstuefe added 2 commits June 20, 2023 17:12

Merge branch 'master' into JDK-8310110-Trace-page-sizes

b05e185

remove test changes from patch

3a84de1

shipilev approved these changes Jun 20, 2023

View reviewed changes

rkennke reviewed Jun 28, 2023

View reviewed changes

feedback roman

8cb392c

shipilev approved these changes Jun 28, 2023

View reviewed changes

rkennke approved these changes Jun 28, 2023

View reviewed changes

openjdk bot added the integrated Pull request has been integrated label Jun 28, 2023

openjdk bot closed this Jun 28, 2023

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Jun 28, 2023

tstuefe mentioned this pull request Jul 14, 2023

8311986: Disable runtime/os/TestTracePageSizes.java for ShenandoahGC #14858

Closed

3 tasks

tstuefe deleted the JDK-8310110-Trace-page-sizes branch July 25, 2023 13:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8310110: Shenandoah: Trace page sizes #14486

8310110: Shenandoah: Trace page sizes #14486

tstuefe commented Jun 15, 2023 •

edited by openjdk bot

Loading

bridgekeeper bot commented Jun 15, 2023

openjdk bot commented Jun 15, 2023

mlbridge bot commented Jun 16, 2023 •

edited

Loading

shipilev left a comment

shipilev commented Jun 16, 2023

tstuefe commented Jun 16, 2023

tstuefe commented Jun 19, 2023

shipilev left a comment

openjdk bot commented Jun 19, 2023 •

edited

Loading

tstuefe commented Jun 19, 2023

shipilev left a comment

tstuefe commented Jun 20, 2023

shipilev commented Jun 20, 2023

tstuefe commented Jun 20, 2023

shipilev commented Jun 20, 2023 •

edited

Loading

tstuefe commented Jun 20, 2023

shipilev left a comment

tstuefe commented Jun 27, 2023

rkennke Jun 28, 2023

tstuefe Jun 28, 2023

rkennke Jun 28, 2023

shipilev Jun 28, 2023

tstuefe Jun 28, 2023

rkennke left a comment

tstuefe commented Jun 28, 2023

openjdk bot commented Jun 28, 2023

openjdk bot commented Jun 28, 2023

8310110: Shenandoah: Trace page sizes #14486

8310110: Shenandoah: Trace page sizes #14486

Conversation

tstuefe commented Jun 15, 2023 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Reviewing

Webrev

bridgekeeper bot commented Jun 15, 2023

openjdk bot commented Jun 15, 2023

mlbridge bot commented Jun 16, 2023 • edited Loading

Webrevs

shipilev left a comment

Choose a reason for hiding this comment

shipilev commented Jun 16, 2023

tstuefe commented Jun 16, 2023

tstuefe commented Jun 19, 2023

shipilev left a comment

Choose a reason for hiding this comment

openjdk bot commented Jun 19, 2023 • edited Loading

tstuefe commented Jun 19, 2023

shipilev left a comment

Choose a reason for hiding this comment

tstuefe commented Jun 20, 2023

shipilev commented Jun 20, 2023

tstuefe commented Jun 20, 2023

shipilev commented Jun 20, 2023 • edited Loading

tstuefe commented Jun 20, 2023

shipilev left a comment

Choose a reason for hiding this comment

tstuefe commented Jun 27, 2023

rkennke Jun 28, 2023

Choose a reason for hiding this comment

tstuefe Jun 28, 2023

Choose a reason for hiding this comment

rkennke Jun 28, 2023

Choose a reason for hiding this comment

shipilev Jun 28, 2023

Choose a reason for hiding this comment

tstuefe Jun 28, 2023

Choose a reason for hiding this comment

rkennke left a comment

Choose a reason for hiding this comment

tstuefe commented Jun 28, 2023

openjdk bot commented Jun 28, 2023

openjdk bot commented Jun 28, 2023

tstuefe commented Jun 15, 2023 •

edited by openjdk bot

Loading

mlbridge bot commented Jun 16, 2023 •

edited

Loading

openjdk bot commented Jun 19, 2023 •

edited

Loading

shipilev commented Jun 20, 2023 •

edited

Loading