8314573: G1: Heap resizing at Remark does not take existing eden regions into account

[tschatzl]

Hi all,

can I have reviews for this change that tries to make the heap resizing policy at Remark more similar to the one after full GC?

The problem is that when calculating the amount of free regions to apply the Min/MaxHeapFreeRatio during Remark g1 does not consider that at Remark we might have already allocated lots of eden regions.
Which means that the heap appears fuller than it would be during full gc, which means that Remark sometimes expands very aggressively due to MinHeapFreeRatio setting. I.e. if at the time of remark pause free regions

There should be no impact wrt to shrinking: MaxHeapFreeRatio is by default (70%) larger than the maximum eden/young gen size (G1MaxNewSizePercent = 60%), so there should be no additional shrinking (that would only lead to additional heap expansion later).

I am aware that fiddling with the values (MaxHeapFreeRatio/G1MaxNewSizePercent) may create a situation where g1 would immediately increase the heap afterwards due to young gen being able to be larger than MaxHeapFreeRatio (if G1MaxNewSizePercent > MaxHeapFreeRatio and G1 would use that large young gen). However this is no different to the current situation after full gc, so I would like to keep it as is. Also for the sake of not complicating this (imo) straightforward change.

A more comprehensive improvement to the problem would be JDK-8238687, where we do the resizing when eden is empty after young gc, but I think this is out of scope here: this fix improves the situation recently reported to me where g1 suddenly started to expand a lot during Remark after upgrades, and should also fix the issue reported in https://bugs.openjdk.org/browse/JDK-6490394 in the last comment (which has been reported around 1 year after the push, which means likely it got overlooked).

Testing: tier1-3, internal perf testing with no changes

Thanks,
Thomas

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8314573: G1: Heap resizing at Remark does not take existing eden regions into account (Bug - P3)

Reviewers

• Kim Barrett (@kimbarrett - Reviewer)
• Albert Mingkun Yang (@albertnetymk - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/15400/head:pull/15400
$ git checkout pull/15400

Update a local copy of the PR:
$ git checkout pull/15400
$ git pull https://git.openjdk.org/jdk.git pull/15400/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 15400

View PR using the GUI difftool:
$ git pr show -t 15400

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/15400.diff

Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back tschatzl! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@tschatzl The following label will be automatically applied to this pull request:

• hotspot-gc

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (9af36cca)

[albertnetymk]

However this is no different to the current situation after full gc

I don't think they are the same -- in full-gc case eden is indeed empty, while in remark-pause the patch treats eden as empty even when it's not.

The version on master treats eden as full and do the heap resizing (expand/shrink). Can we permit only shrink in remark-pause, since eden liveness is unknown at that moment? (Remark-pause occurs not due to alloc-failure, so it is kind of logical not to expand the heap.)

[tschatzl]

However this is no different to the current situation after full gc
I don't think they are the same -- in full-gc case eden is indeed empty, while in remark-pause the patch treats eden as empty even when it's not.

The version on master treats eden as full and do the heap resizing (expand/shrink).

The marking cycle should be a replacement a full collection (or that is the idea), including heap resizing.
Remark pause is just some place in this replacement full collection cycle where the amount of free regions is maximal (given we free empty regions there).

Eden allocations are in some way as if they did not exist from a full collection heap resizing POV. This change implements this (to some degree).

Can we permit only shrink in remark-pause, since eden liveness is unknown at that moment?

We could do an approximation, but for the sake of simplicity this change does not.

(Remark-pause occurs not due to alloc-failure, so it is kind of logical not to expand the heap.)

I do not follow this. When to resize the heap is arbitrary, but it seems to be the "best" place to simulate a full collection cycle.

[kimbarrett]

Looks good, per the problem description.

Seems like "used_after_gc" is a confusing name though. Maybe fix that later, unless you
have a better idea right now (I don't).

[openjdk]

@tschatzl This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8314573: G1: Heap resizing at Remark does not take existing eden regions into account

Reviewed-by: kbarrett, ayang

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 92 new commits pushed to the master branch:

• 725ec0c: 8315020: The macro definition for LoongArch64 zero build is not accurate.
• 1c3177e: 8315029: [BACKOUT] Generational ZGC: Tests crash with assert(index == 0 || is_power_of_2(index))
• dd23f7d: 8315039: Parallel: Remove unimplemented PSYoungGen::oop_iterate
• 5c4f1dc: 8314513: [IR Framework] Some internal IR Framework tests are failing after JDK-8310308 on PPC and Cascade Lake
• cf2d33c: 8299658: C1 compilation crashes in LinearScan::resolve_exception_edge
• 1664e79: 8311792: java/net/httpclient/ResponsePublisher.java fails intermittently with AssertionError: Found some outstanding operations
• 0901d75: 8314762: Make {@Incubating} conventional
• 12de9b0: 8314148: Fix variable scope in SunMSCAPI
• 7fbad4c: 8310596: Utilize existing method frame::interpreter_frame_monitor_size_in_bytes()
• 752a438: 8314684: Add overview docs to loaderConstraints.cpp
• ... and 82 more: https://git.openjdk.org/jdk/compare/33d5dfdab3098549366088c43797bdcaebe02cd6...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[albertnetymk]

Had some offline discussion with Thomas -- this treating-eden-as-free can lead to over-shrinking if using non-default flag values, but that may be a less severe problem than over-expansion.

(Allowing only shrink in Remark is another option, but it would be a larger diff.)

[tschatzl]

Thanks @albertnetymk @kimbarrett for your reviews

/integrate

[openjdk]

Going to push as commit 762b652.
Since your change was applied there have been 110 commits pushed to the master branch:

• a4e97aa: 8314753: Remove support for @beaninfo, @todo, @since.unbundled, and @note
• 5cc64cc: 8309463: IGV: Dynamic graph layout algorithm
• 1d02507: 8314476: TestJstatdPortAndServer.java failed with "java.rmi.NoSuchObjectException: no such object in table"
• 93188bd: 8314569: (fs) Improve normalization of UnixPath for input with trailing slashes
• 8b8182d: 8315087: G1: Use uint for G1 flags indicating percentage
• 3dc266c: 8315070: RISC-V: Clean up platform dependent inline headers
• 25f5df2: 8315073: Zero build on macOS fails after JDK-8303852
• a03954e: 8309697: [TESTBUG] Remove "@requires vm.flagless" from jtreg vectorization tests
• e5ea9aa: 8312570: [TESTBUG] Jtreg compiler/loopopts/superword/TestDependencyOffsets.java fails on 512-bit SVE
• 1cb2cc6: 8308464: Shared array class should not always be loaded in boot loader
• ... and 100 more: https://git.openjdk.org/jdk/compare/33d5dfdab3098549366088c43797bdcaebe02cd6...master

Your commit was automatically rebased without conflicts.

[openjdk]

@tschatzl Pushed as commit 762b652.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

[mmyxym]

/backport jdk21u-dev

[openjdk]

@mmyxym the backport was successfully created on the branch backport-mmyxym-762b6529 in my personal fork of openjdk/jdk21u-dev. To create a pull request with this backport targeting openjdk/jdk21u-dev:master, just click the following link:

➡️ Create pull request

The title of the pull request is automatically filled in correctly and below you find a suggestion for the pull request body:

Hi all,

This pull request contains a backport of commit 762b6529 from the openjdk/jdk repository.

The commit being backported was authored by Thomas Schatzl on 29 Aug 2023 and was reviewed by Kim Barrett and Albert Mingkun Yang.

Thanks!

If you need to update the source branch of the pull then run the following commands in a local clone of your personal fork of openjdk/jdk21u-dev:

$ git fetch https://github.com/openjdk-bots/jdk21u-dev.git backport-mmyxym-762b6529:backport-mmyxym-762b6529
$ git checkout backport-mmyxym-762b6529
# make changes
$ git add paths/to/changed/files
$ git commit --message 'Describe additional changes made'
$ git push https://github.com/openjdk-bots/jdk21u-dev.git backport-mmyxym-762b6529