8255984: Shenandoah: "adaptive" heuristic is prone to missing load spikes

[earthling-amzn]

This change adds a "reactive" heuristic for triggering concurrent GC cycles.

The reactive heuristic maintains a margin of error and an allocation spike detection mechanism to trigger cycles somewhat more aggressively than the 'adaptive' heuristic. This heuristic 'reacts' to the outcome of GC cycles by adjusting the sensitivity of the triggers.

JBS ticket is here: https://bugs.openjdk.java.net/browse/JDK-8255984

The "adaptive" heuristic remains the default.

Steps to reproduce and test will follow shortly (there are no new jtreg test failures for Shenandoah with this change).

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed

Testing

	Linux aarch64	Linux arm	Linux ppc64le	Linux s390x	Linux x64	Linux x86	Windows x64	macOS x64
Build	✔️ (1/1 passed)	✔️ (1/1 passed)	✔️ (1/1 passed)	✔️ (1/1 passed)	✔️ (6/6 passed)	✔️ (2/2 passed)	✔️ (2/2 passed)	✔️ (2/2 passed)
Test (tier1)					✔️ (9/9 passed)	⏳ (1/9 running)	⏳ (6/9 running)	✔️ (9/9 passed)

Issue

• JDK-8255984: Shenandoah: "adaptive" heuristic is prone to missing load spikes

Reviewers

• Aleksey Shipilev (@shipilev - Reviewer) ⚠️ Review applies to 8b58ec8

Download

$ git fetch https://git.openjdk.java.net/jdk pull/1099/head:pull/1099
$ git checkout pull/1099

[bridgekeeper]

Hi @earthling-amzn, welcome to this OpenJDK project and thanks for contributing!

We do not recognize you as Contributor and need to ensure you have signed the Oracle Contributor Agreement (OCA). If you have not signed the OCA, please follow the instructions. Please fill in your GitHub username in the "Username" field of the application. Once you have signed the OCA, please let us know by writing /signed in a comment in this pull request.

If you already are an OpenJDK Author, Committer or Reviewer, please click here to open a new issue so that we can record that fact. Please use "Add GitHub user earthling-amzn" as summary for the issue.

If you are contributing this work on behalf of your employer and your employer has signed the OCA, please let us know by writing /covered in a comment in this pull request.

[openjdk]

@earthling-amzn The following labels will be automatically applied to this pull request:

• hotspot-gc
• shenandoah

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[earthling-amzn]

/covered

[bridgekeeper]

Thank you! Please allow for a few business days to verify that your employer has signed the OCA. Also, please note that pull requests that are pending an OCA check will not usually be evaluated, so your patience is appreciated!

[robilad]

Hi,

Please contact me at Dalibor.topic@oracle.com so that I can verify your GitHub account.

Thanks!

[shipilev]

This looks interesting. I have to look closer at this, but maybe we should instead fold this to adaptive. This would also resolve the testing question: Shenandoah tests usually run with most (all?) heuristics, and so new heuristics needs to be followed with lots of test changes. Folding this to adaptive implicitly fixes that.

[earthling-amzn]

I think folding it in to adaptive makes sense. I went the conservative approach originally just to keep the changes isolated. I'll prepare a commit with these changes built directly in the adaptive heuristic.

[shipilev]

While waiting for merge to "adaptive", here is a round of minor things from the first read.

[shipilev]

Thanks for updates. I would like to dive in with some testing. Second read nits below, meanwhile.

[shipilev]

Oh, and change PR title to "8255984: Shenandoah: "adaptive" heuristic is prone to missing load spikes", so that bots know what issue it refers to.

[mlbridge]

Webrevs

• 07: Full - Incremental (72f15a6)
• 06: Full - Incremental (8b58ec8)
• 05: Full - Incremental (68ba628)
• 04: Full - Incremental (f6bb039)
• 03: Full - Incremental (3ec8f3c)
• 02: Full - Incremental (6ef8dc6)
• 01: Full - Incremental (275e80a)
• 00: Full (7f1de19)

[earthling-amzn]

Though we tested this with several benchmarks and internal services, these changes were primarily developed using the HyperAlloc benchmark. The attached chart shows stacked histograms for the heuristics that triggered a collection cycle during a run of HyperAlloc.

Each test run was given a 3g heap. HyperAlloc targeted a 1g/s allocation rate (-a 1000) with a max object size of 1000 bytes (-x 1000). It was configured to maintain 1gb of live objects on the heap (-s 1024) and it ran for two minutes (-d 120). Each row in the chart is for a different allocation smoothness factor. The top row has an allocation smoothness of zero (i.e., most spiky), the bottom row has an allocation smoothness of 0.4, still spiky, but manageable. Each column in the chart is a different heuristic. On the left is the original adaptive heuristic (with default parameters). On the far right is the compact heuristic. In the middle are runs of the original reactive heuristic with different sensitivities to allocations spikes. The purple bars are allocation failures. The light blue bars are instances where an allocation spike was detected and triggered a cycle.

[shipilev]

The more I study this patch, the more I like it, good job. Another round of changes below. I fixed a few mentioned issues in my private working copy while doing initial performance testing. Would be good to have an updated PR for testing too.

[shipilev]

Yes, it mostly matches what I would expect from this heuristic improvement. As it stands right now, "adaptive" balances way too closely to the cliff, and so it is expected that improvement would make GC cycles more frequent to move GC triggers away from it. On the chart, "Headroom" probably means "Average allocation rate" trigger? It still a bit sad we cannot handle smoothness=0.0, but that probably requires future-predicting capabilities not yet available at current technology level.

[shipilev]

I don't understand why GH actions do not test your patch :/ Can you go to https://github.com/earthling-amzn/jdk/settings/ and check that both "Actions" and "Secrets" do not have anything special defined? "Actions" should say "Allow all actions", "Secrets" should not have JDK_SUBMIT_PLATFORMS defined.

[earthling-amzn]

"Allow all actions" is checked, nothing defined for secrets. Is it because the title didn't have the JBS issue number in it when I opened the PR?

[shipilev]

Unlikely. If you go to https://github.com/earthling-amzn/jdk/actions, can you trigger the workflow by hand?

[earthling-amzn]

Ah, it says this at the actions link:

Workflows aren’t being run on this forked repository
Because this repository contained workflow files when it was forked, we have disabled them from running on this fork. Make sure you understand the configured workflows and their expected usage before enabling Actions on this repository.

I've re-enabled them and the pre-submit action is running now.

[earthling-amzn]

I'm also running my own test suite (heapotheysis/HyperAlloc) with different permutations for the margin of error, spike threshold and sample frequency. I'll run the whole thing again without any headroom buffer. Will have the results by end of day.

[earthling-amzn]

Finished running more tests. My preference is still to keep the headroom/penalties in place. It can run without these, but there is a modest increase in degenerated cycles. If we did remove these safeguards, I'd recommend increasing the default sample frequency to 100hz.

[shipilev]

Okay then! Time to integrate and do anything else in follow-ups.

[openjdk]

@earthling-amzn This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8255984: Shenandoah: "adaptive" heuristic is prone to missing load spikes

Reviewed-by: shade

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 114 new commits pushed to the master branch:

• fa75ad6: 8256725: Metaspace: better blocktree and binlist asserts
• 9de5d09: 8256585: Remove in-place conversion vector operators from Vector API
• 18e8506: 8256860: S390 builds broken after JDK-8254231
• 042734c: 8251549: Update docs on building for Git
• ae0ca74: 8247432: Update IANA Language Subtag Registry to Version 2020-09-29
• 8ddf5e1: 8256809: Annotation processing causes NPE during flow analysis
• aabc9ca: 8255678: Add Mutex::try_lock version without rank checks
• 884b9ff: 8256485: Zero VM build broken after JDK-8254231
• 84429cd: 8249144: Potential memory leak in TypedMethodOptionMatcher
• 895cc4f: 8255675: Typo in java.net.HttpURLConnection
• ... and 104 more: https://git.openjdk.java.net/jdk/compare/9efbb463a4ad7c24f28600e231d65d14d38c9af2...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@shipilev) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

[earthling-amzn]

Sorry, added one bug fix and another small change. The bug caused the average allocation rate to be much higher than it should be. The other change is to only consider the values which are actually sampled as a "spike" (as opposed to previously when it would recompute the instantaneous rate on every call to should_start_gc). For what it's worth, I made these changes while the headroom/penalty adjustments were disabled. I'm more comfortable with taking those adjustments out now.

[shipilev]

Okay, I have a minor suggestion that seems still preserve the semantics. It is generally looks good, the testing passes, and benchmarks look okay (some regressions, but expected). Feel free to integrate.

[earthling-amzn]

/integrate

[openjdk]

@earthling-amzn
Your change (at version 72f15a6) is now ready to be sponsored by a Committer.

[shipilev]

/sponsor

[shipilev]

Congratulations! If you have time, please work on removing old alloc spike threshold and penalties handling.

[openjdk]

@shipilev @earthling-amzn Since your change was applied there have been 114 commits pushed to the master branch:

• fa75ad6: 8256725: Metaspace: better blocktree and binlist asserts
• 9de5d09: 8256585: Remove in-place conversion vector operators from Vector API
• 18e8506: 8256860: S390 builds broken after JDK-8254231
• 042734c: 8251549: Update docs on building for Git
• ae0ca74: 8247432: Update IANA Language Subtag Registry to Version 2020-09-29
• 8ddf5e1: 8256809: Annotation processing causes NPE during flow analysis
• aabc9ca: 8255678: Add Mutex::try_lock version without rank checks
• 884b9ff: 8256485: Zero VM build broken after JDK-8254231
• 84429cd: 8249144: Potential memory leak in TypedMethodOptionMatcher
• 895cc4f: 8255675: Typo in java.net.HttpURLConnection
• ... and 104 more: https://git.openjdk.java.net/jdk/compare/9efbb463a4ad7c24f28600e231d65d14d38c9af2...master

Your commit was automatically rebased without conflicts.

Pushed as commit aac5c2a.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.