8350148: Native stack overflow when writing Java heap objects into AOT cache

[iklam]

Please review this patch that fixes a problem that was found in the Leyden repo: when finding all cacheable heap objects with recursive calls to HeapShared::archive_reachable_objects_from(), very deep reference chains could cause overflow on the native stack.

The fix is to do the recursion using a side table, without making any recursive calls.

Note: the kind of deep reference chains do not seem to happen with the mainline. It happened in the Leyden repo only after we enabled the caching of WeakReference objects, which are not cacheable in the mainline.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8350148: Native stack overflow when writing Java heap objects into AOT cache (Bug - P3)

Reviewers

• Igor Veresov (@veresov - Reviewer)
• Matias Saavedra Silva (@matias9927 - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/23654/head:pull/23654
$ git checkout pull/23654

Update a local copy of the PR:
$ git checkout pull/23654
$ git pull https://git.openjdk.org/jdk.git pull/23654/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 23654

View PR using the GUI difftool:
$ git pr show -t 23654

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/23654.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back iklam! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@iklam This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8350148: Native stack overflow when writing Java heap objects into AOT cache

Reviewed-by: iveresov, matsaave

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 240 new commits pushed to the master branch:

• 5cd4fe6: 8348309: MultiNST tests need more debugging and timing
• 7c22b81: 8350811: [JMH] test foreign.StrLenTest failed with StringIndexOutOfBoundsException for size=451
• 54fe643: 8347433: Deprecate XML interchange in java.management/javax/management/modelmbean/DescriptorSupport for removal
• 155697f: 8349623: [ASAN] Gtest os_linux.glibc_mallinfo_wrapper_vm fails
• 4066f33: 8350565: NMT: remaining memory flag/type to be replaced with memory tag
• 7314efc: 8351377: Fix the ProblemList for com/sun/management/OperatingSystemMXBean cpuLoad tests on AIX
• 7c9a784: 8351223: Update localized resources in keytool and jarsigner
• a23fb0a: 8348110: Update LCMS to 2.17
• 0892913: 8351086: (fc) Make java/nio/channels/FileChannel/BlockDeviceSize.java test manual
• 93f2260: 8351294: (fs) Minor verbiage correction for Files.createTemp{Directory,File}
• ... and 230 more: https://git.openjdk.org/jdk/compare/7d11418c820b46926a25907766d16083a4b349de...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[openjdk]

@iklam The following label will be automatically applied to this pull request:

• hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (4bdb21b0)

[iklam]

@shipilev @rose00 could you take a look?

[veresov]

Looks good

[veresov]

Suggested change

	class HeapShared::ReferentPusher: public BasicOopIterateClosure {
	class HeapShared::ReferentPusher : public BasicOopIterateClosure {

[matias9927]

LGTM!

[iklam]

Thanks @veresov and @matias9927 for the review
/integrate

[openjdk]

Going to push as commit 8a5ed47.
Since your change was applied there have been 272 commits pushed to the master branch:

• 5928209: 8347405: MergeStores with reverse bytes order value
• f984c2b: 8351505: (fs) Typo in the documentation of java.nio.file.spi.FileSystemProvider.getFileSystem()
• ffa6340: 8351567: Jar Manifest test ValueUtf8Coding produces misleading diagnostic output
• 8d8bd0c: 8349492: Update sun/security/pkcs12/KeytoolOpensslInteropTest.java to use a recent Openssl version
• 73465b9: 8160327: Support for thumbnails present in APP1 marker for JPEG
• dbdbbd4: 8348597: Update HarfBuzz to 10.4.0
• 7999091: 8351555: Help section added in JDK-8350638 uses invalid HTML
• 8450ae9: 8351440: Link with -reproducible on macOS
• b40be22: 8333393: PhaseCFG::insert_anti_dependences can fail to raise LCAs and to add necessary anti-dependence edges
• 6b84bde: 8350007: Add usage message to the javadoc executable
• ... and 262 more: https://git.openjdk.org/jdk/compare/7d11418c820b46926a25907766d16083a4b349de...master

Your commit was automatically rebased without conflicts.

[openjdk]

@iklam Pushed as commit 8a5ed47.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.