8314021: HeapDump: Optimize segmented heap file merging phase #15245

y1yang0 · 2023-08-11T09:31:56Z

This patch reduce ~16%(24s->20s) pahse 2 merge time during dumping 32g heap with 96threads and fixes a memory leak of compressor

You might argue why this is Linux-only optimization, because sendfile requires at least socket fd in other platforms(aix sendfile maxos sendfile win32 TransmitFile), while only Linux supports both two file descriptors.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8314021: HeapDump: Optimize segmented heap file merging phase (Enhancement - P4)

Reviewers

Alex Menkov (@alexmenkov - Reviewer)
Kevin Walls (@kevinjwalls - Committer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/15245/head:pull/15245
$ git checkout pull/15245

Update a local copy of the PR:
$ git checkout pull/15245
$ git pull https://git.openjdk.org/jdk.git pull/15245/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 15245

View PR using the GUI difftool:
$ git pr show -t 15245

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/15245.diff

Webrev

Link to Webrev Comment

bridgekeeper · 2023-08-11T09:32:58Z

👋 Welcome back yyang! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2023-08-11T09:34:41Z

@y1yang0 The following label will be automatically applied to this pull request:

hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

y1yang0 · 2023-08-11T09:40:58Z

/label remove hotspot-runtime
/label add serviceability

openjdk · 2023-08-11T09:42:28Z

@y1yang0
The hotspot-runtime label was successfully removed.

openjdk · 2023-08-11T09:43:04Z

@y1yang0
The serviceability label was successfully added.

mlbridge · 2023-08-11T09:49:28Z

Webrevs

01: Full - Incremental (0eb02c14)
00: Full (ba381aaf)

alexmenkov · 2023-09-07T00:56:40Z

The fix looks good to me in general, but I'm not sure about code organization.
src/hotspot/share/runtime/os.hpp describes rules for os* files.
It states:
// Platform-independent source files should not include these header files
// (although sadly there are some rare exceptions ...)
And the change adds one more exception.
I'd like to hear runtime guys opinion.

alexmenkov · 2023-09-07T00:49:14Z

src/hotspot/share/services/heapDumper.cpp

+#ifdef LINUX
+      // Merge segmented heap files via sendfile, it's more efficient than the
+      // read+write combination, which would require transferring data to and from
+      // user space.
+      merge_file_fast(path);


Looks like you don't need separate method for linux.
Would it be better to make linux-specific implementation of merge_file():
#ifdef LINUX
// Merge segmented heap files via sendfile
void DumpMerger::merge_file(char* path) {
...
}
#else
// Generic implementation using read+write
void DumpMerger::merge_file(char* path) {
...
}
#endif

y1yang0 · 2023-09-07T02:13:21Z

The fix looks good to me in general, but I'm not sure about code organization. src/hotspot/share/runtime/os.hpp describes rules for os* files. It states: // Platform-independent source files should not include these header files // (although sadly there are some rare exceptions ...) And the change adds one more exception. I'd like to hear runtime guys opinion.

Thanks for the review, yes, grepping os_linux.hpp found several exceptions, I'd like to see if runtime folks have alternatives, too.

/label add hotspot-runtime

openjdk · 2023-09-07T02:14:07Z

@y1yang0
The hotspot-runtime label was successfully added.

y1yang0 · 2023-09-11T02:24:58Z

Hi @alexmenkov, can we move on since there are no more comments from runtime folks. We can still propose a follow-up change if someone have strong objection/concern on this "exception"

openjdk · 2023-09-12T00:16:37Z

@y1yang0 This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8314021: HeapDump: Optimize segmented heap file merging phase

Reviewed-by: amenkov, kevinw

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 488 new commits pushed to the master branch:

3828dc9: 8314236: Overflow in Collections.rotate
1203e11: 8294969: Convert jdk.jdeps javap to use the Classfile API
fbc766e: 8315988: Parallel: Make TestAggressiveHeap use createTestJvm
aa0ebee: 8316341: sun/security/pkcs11/PKCS11Test.java needs adjustment on Linux ppc64le Ubuntu 22
ca3fe7b: 8315932: runtime/InvocationTests spend a lot of time on dependency verification
f440329: 8316391: (zipfs) ZipFileSystem.readFullyAt does not tolerate short reads
4b8f5d0: 8316273: JDK-8315818 broke JVMCIPrintProperties on libgraal
e624198: 8316387: Exclude more failing multicast tests on AIX after JDK-8315651
c92bdb0: 8316187: Modernize examples in StringTokenizer and {Date,Number}Format
8f46abc: 8315889: Open source several Swing HTMLDocument related tests
... and 478 more: https://git.openjdk.org/jdk/compare/31a307f2fbe7b99435f50e5404c2a95f07b9a77b...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

y1yang0 · 2023-09-18T02:40:40Z

Can I have a second review? Maybe @plummercj @kevinjwalls

kevinjwalls

Yes, looks good to me.
Apologies for delay, had looked at this previously but not come back for final review.

y1yang0 · 2023-09-19T02:01:54Z

Apologies for delay, had looked at this previously but not come back for final review.

Never mind:) Thanks @alexmenkov and @kevinjwalls for the reviews.

/integrate

openjdk · 2023-09-19T02:03:19Z

Going to push as commit 3760a04.
Since your change was applied there have been 500 commits pushed to the master branch:

f226ceb: 8316440: LambdasInTwoArchives.java failed to find WhiteBox.class
24c3d86: 8315804: Open source several Swing JTabbedPane JTextArea JTextField tests
373e37b: 8313813: Field sun.util.calendar.CalendarDate#forceStandardTime is never set
5308bbb: 8246280: Refine API to model sealed classes and interfaces in javax.lang.model
dcea9bf: 8301247: JPackage app-image exe launches multiple exe's in JDK 17+
1b104b6: 8314909: tools/jpackage/windows/Win8282351Test.java fails with java.lang.AssertionError: Expected [0]. Actual [1618]:
4421951: 8316142: Enable parallelism in vmTestbase/nsk/monitoring/stress/lowmem tests
f09b7af: 8311220: Optimization for StringLatin UpperLower
2e2d49c: 8316400: Exclude jdk/jfr/event/runtime/TestResidentSetSizeEvent.java on AIX
bf9d1e2: 8316399: Exclude java/net/MulticastSocket/Promiscuous.java on AIX
... and 490 more: https://git.openjdk.org/jdk/compare/31a307f2fbe7b99435f50e5404c2a95f07b9a77b...master

Your commit was automatically rebased without conflicts.

openjdk · 2023-09-19T02:03:27Z

@y1yang0 Pushed as commit 3760a04.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

dholmes-ora · 2023-09-21T01:18:19Z

I had not noticed this PR sorry. I do not like the fact we have Linux-only functionality being added with no intent to supply similar functionality on other platforms. I also do not like the fact we had to ifdef the Linux code into the shared code. It would have been metter to place the code in os::Linux and then have a single LINUX_ONLY() to make that call.

y1yang0 · 2023-09-21T02:12:06Z

I had not noticed this PR sorry. I do not like the fact we have Linux-only functionality being added with no intent to supply similar functionality on other platforms. I also do not like the fact we had to ifdef the Linux code into the shared code. It would have been metter to place the code in os::Linux and then have a single LINUX_ONLY() to make that call.

Okay, I'll prepare a follow-up PR to do this once #15843 merged.

y1yang0 · 2023-10-11T06:49:30Z

I had not noticed this PR sorry. I do not like the fact we have Linux-only functionality being added with no intent to supply similar functionality on other platforms. I also do not like the fact we had to ifdef the Linux code into the shared code. It would have been metter to place the code in os::Linux and then have a single LINUX_ONLY() to make that call.

I made a simple attempt, but it is difficult to integrate this portion of code into OS Linux because it requires modifying the writer's written bytes and setting error messages. Logically, it belongs to DumpMerge. Abstracting it into a function similar to concatenate_file and placing it into OS Linux seems challenging.

dholmes-ora · 2023-10-19T01:58:25Z

I think it can be improved somewhat even if we can't simply pass the _writer through.

8314021: HeapDump: Optimize segmented heap file merging phase

ba381aa

openjdk bot added the rfr Pull request is ready for review label Aug 11, 2023

openjdk bot added the hotspot-runtime hotspot-runtime-dev@openjdk.org label Aug 11, 2023

openjdk bot removed the hotspot-runtime hotspot-runtime-dev@openjdk.org label Aug 11, 2023

openjdk bot added the serviceability serviceability-dev@openjdk.org label Aug 11, 2023

alexmenkov reviewed Sep 7, 2023

View reviewed changes

one merge_file for all

0eb02c1

openjdk bot added the hotspot-runtime hotspot-runtime-dev@openjdk.org label Sep 7, 2023

alexmenkov approved these changes Sep 12, 2023

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Sep 12, 2023

kevinjwalls approved these changes Sep 18, 2023

View reviewed changes

openjdk bot added the integrated Pull request has been integrated label Sep 19, 2023

openjdk bot closed this Sep 19, 2023

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Sep 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8314021: HeapDump: Optimize segmented heap file merging phase #15245

8314021: HeapDump: Optimize segmented heap file merging phase #15245

y1yang0 commented Aug 11, 2023 •

edited by openjdk bot

Loading

bridgekeeper bot commented Aug 11, 2023

openjdk bot commented Aug 11, 2023

y1yang0 commented Aug 11, 2023

openjdk bot commented Aug 11, 2023

openjdk bot commented Aug 11, 2023

mlbridge bot commented Aug 11, 2023 •

edited

Loading

alexmenkov commented Sep 7, 2023

alexmenkov Sep 7, 2023

y1yang0 commented Sep 7, 2023 •

edited

Loading

openjdk bot commented Sep 7, 2023

y1yang0 commented Sep 11, 2023

openjdk bot commented Sep 12, 2023 •

edited

Loading

y1yang0 commented Sep 18, 2023

kevinjwalls left a comment

y1yang0 commented Sep 19, 2023

openjdk bot commented Sep 19, 2023

openjdk bot commented Sep 19, 2023

dholmes-ora commented Sep 21, 2023

y1yang0 commented Sep 21, 2023 •

edited

Loading

y1yang0 commented Oct 11, 2023

dholmes-ora commented Oct 19, 2023

8314021: HeapDump: Optimize segmented heap file merging phase #15245

8314021: HeapDump: Optimize segmented heap file merging phase #15245

Conversation

y1yang0 commented Aug 11, 2023 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Reviewing

Webrev

bridgekeeper bot commented Aug 11, 2023

openjdk bot commented Aug 11, 2023

y1yang0 commented Aug 11, 2023

openjdk bot commented Aug 11, 2023

openjdk bot commented Aug 11, 2023

mlbridge bot commented Aug 11, 2023 • edited Loading

Webrevs

alexmenkov commented Sep 7, 2023

alexmenkov Sep 7, 2023

Choose a reason for hiding this comment

y1yang0 commented Sep 7, 2023 • edited Loading

openjdk bot commented Sep 7, 2023

y1yang0 commented Sep 11, 2023

openjdk bot commented Sep 12, 2023 • edited Loading

y1yang0 commented Sep 18, 2023

kevinjwalls left a comment

Choose a reason for hiding this comment

y1yang0 commented Sep 19, 2023

openjdk bot commented Sep 19, 2023

openjdk bot commented Sep 19, 2023

dholmes-ora commented Sep 21, 2023

y1yang0 commented Sep 21, 2023 • edited Loading

y1yang0 commented Oct 11, 2023

dholmes-ora commented Oct 19, 2023

y1yang0 commented Aug 11, 2023 •

edited by openjdk bot

Loading

mlbridge bot commented Aug 11, 2023 •

edited

Loading

y1yang0 commented Sep 7, 2023 •

edited

Loading

openjdk bot commented Sep 12, 2023 •

edited

Loading

y1yang0 commented Sep 21, 2023 •

edited

Loading