Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8314021: HeapDump: Optimize segmented heap file merging phase #15245

Closed
wants to merge 2 commits into from

Conversation

y1yang0
Copy link
Member

@y1yang0 y1yang0 commented Aug 11, 2023

This patch reduce ~16%(24s->20s) pahse 2 merge time during dumping 32g heap with 96threads and fixes a memory leak of compressor

You might argue why this is Linux-only optimization, because sendfile requires at least socket fd in other platforms(aix sendfile maxos sendfile win32 TransmitFile), while only Linux supports both two file descriptors.


Progress

  • Change must be properly reviewed (1 review required, with at least 1 Reviewer)
  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue

Issue

  • JDK-8314021: HeapDump: Optimize segmented heap file merging phase (Enhancement - P4)

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/15245/head:pull/15245
$ git checkout pull/15245

Update a local copy of the PR:
$ git checkout pull/15245
$ git pull https://git.openjdk.org/jdk.git pull/15245/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 15245

View PR using the GUI difftool:
$ git pr show -t 15245

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/15245.diff

Webrev

Link to Webrev Comment

@bridgekeeper
Copy link

bridgekeeper bot commented Aug 11, 2023

👋 Welcome back yyang! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk openjdk bot added the rfr Pull request is ready for review label Aug 11, 2023
@openjdk
Copy link

openjdk bot commented Aug 11, 2023

@y1yang0 The following label will be automatically applied to this pull request:

  • hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added the hotspot-runtime hotspot-runtime-dev@openjdk.org label Aug 11, 2023
@y1yang0
Copy link
Member Author

y1yang0 commented Aug 11, 2023

/label remove hotspot-runtime
/label add serviceability

@openjdk openjdk bot removed the hotspot-runtime hotspot-runtime-dev@openjdk.org label Aug 11, 2023
@openjdk
Copy link

openjdk bot commented Aug 11, 2023

@y1yang0
The hotspot-runtime label was successfully removed.

@openjdk openjdk bot added the serviceability serviceability-dev@openjdk.org label Aug 11, 2023
@openjdk
Copy link

openjdk bot commented Aug 11, 2023

@y1yang0
The serviceability label was successfully added.

@mlbridge
Copy link

mlbridge bot commented Aug 11, 2023

Webrevs

@alexmenkov
Copy link

The fix looks good to me in general, but I'm not sure about code organization.
src/hotspot/share/runtime/os.hpp describes rules for os* files.
It states:
// Platform-independent source files should not include these header files
// (although sadly there are some rare exceptions ...)
And the change adds one more exception.
I'd like to hear runtime guys opinion.

Comment on lines 1647 to 1651
#ifdef LINUX
// Merge segmented heap files via sendfile, it's more efficient than the
// read+write combination, which would require transferring data to and from
// user space.
merge_file_fast(path);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you don't need separate method for linux.
Would it be better to make linux-specific implementation of merge_file():
#ifdef LINUX
// Merge segmented heap files via sendfile
void DumpMerger::merge_file(char* path) {
...
}
#else
// Generic implementation using read+write
void DumpMerger::merge_file(char* path) {
...
}
#endif

@y1yang0
Copy link
Member Author

y1yang0 commented Sep 7, 2023

The fix looks good to me in general, but I'm not sure about code organization. src/hotspot/share/runtime/os.hpp describes rules for os* files. It states: // Platform-independent source files should not include these header files // (although sadly there are some rare exceptions ...) And the change adds one more exception. I'd like to hear runtime guys opinion.

Thanks for the review, yes, grepping os_linux.hpp found several exceptions, I'd like to see if runtime folks have alternatives, too.

/label add hotspot-runtime

@openjdk openjdk bot added the hotspot-runtime hotspot-runtime-dev@openjdk.org label Sep 7, 2023
@openjdk
Copy link

openjdk bot commented Sep 7, 2023

@y1yang0
The hotspot-runtime label was successfully added.

@y1yang0
Copy link
Member Author

y1yang0 commented Sep 11, 2023

Hi @alexmenkov, can we move on since there are no more comments from runtime folks. We can still propose a follow-up change if someone have strong objection/concern on this "exception"

@openjdk
Copy link

openjdk bot commented Sep 12, 2023

@y1yang0 This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8314021: HeapDump: Optimize segmented heap file merging phase

Reviewed-by: amenkov, kevinw

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 488 new commits pushed to the master branch:

  • 3828dc9: 8314236: Overflow in Collections.rotate
  • 1203e11: 8294969: Convert jdk.jdeps javap to use the Classfile API
  • fbc766e: 8315988: Parallel: Make TestAggressiveHeap use createTestJvm
  • aa0ebee: 8316341: sun/security/pkcs11/PKCS11Test.java needs adjustment on Linux ppc64le Ubuntu 22
  • ca3fe7b: 8315932: runtime/InvocationTests spend a lot of time on dependency verification
  • f440329: 8316391: (zipfs) ZipFileSystem.readFullyAt does not tolerate short reads
  • 4b8f5d0: 8316273: JDK-8315818 broke JVMCIPrintProperties on libgraal
  • e624198: 8316387: Exclude more failing multicast tests on AIX after JDK-8315651
  • c92bdb0: 8316187: Modernize examples in StringTokenizer and {Date,Number}Format
  • 8f46abc: 8315889: Open source several Swing HTMLDocument related tests
  • ... and 478 more: https://git.openjdk.org/jdk/compare/31a307f2fbe7b99435f50e5404c2a95f07b9a77b...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready Pull request is ready to be integrated label Sep 12, 2023
@y1yang0
Copy link
Member Author

y1yang0 commented Sep 18, 2023

Can I have a second review? Maybe @plummercj @kevinjwalls

Copy link
Contributor

@kevinjwalls kevinjwalls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, looks good to me.
Apologies for delay, had looked at this previously but not come back for final review.

@y1yang0
Copy link
Member Author

y1yang0 commented Sep 19, 2023

Apologies for delay, had looked at this previously but not come back for final review.

Never mind:) Thanks @alexmenkov and @kevinjwalls for the reviews.

/integrate

@openjdk
Copy link

openjdk bot commented Sep 19, 2023

Going to push as commit 3760a04.
Since your change was applied there have been 500 commits pushed to the master branch:

  • f226ceb: 8316440: LambdasInTwoArchives.java failed to find WhiteBox.class
  • 24c3d86: 8315804: Open source several Swing JTabbedPane JTextArea JTextField tests
  • 373e37b: 8313813: Field sun.util.calendar.CalendarDate#forceStandardTime is never set
  • 5308bbb: 8246280: Refine API to model sealed classes and interfaces in javax.lang.model
  • dcea9bf: 8301247: JPackage app-image exe launches multiple exe's in JDK 17+
  • 1b104b6: 8314909: tools/jpackage/windows/Win8282351Test.java fails with java.lang.AssertionError: Expected [0]. Actual [1618]:
  • 4421951: 8316142: Enable parallelism in vmTestbase/nsk/monitoring/stress/lowmem tests
  • f09b7af: 8311220: Optimization for StringLatin UpperLower
  • 2e2d49c: 8316400: Exclude jdk/jfr/event/runtime/TestResidentSetSizeEvent.java on AIX
  • bf9d1e2: 8316399: Exclude java/net/MulticastSocket/Promiscuous.java on AIX
  • ... and 490 more: https://git.openjdk.org/jdk/compare/31a307f2fbe7b99435f50e5404c2a95f07b9a77b...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot added the integrated Pull request has been integrated label Sep 19, 2023
@openjdk openjdk bot closed this Sep 19, 2023
@openjdk openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Sep 19, 2023
@openjdk
Copy link

openjdk bot commented Sep 19, 2023

@y1yang0 Pushed as commit 3760a04.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

@dholmes-ora
Copy link
Member

I had not noticed this PR sorry. I do not like the fact we have Linux-only functionality being added with no intent to supply similar functionality on other platforms. I also do not like the fact we had to ifdef the Linux code into the shared code. It would have been metter to place the code in os::Linux and then have a single LINUX_ONLY() to make that call.

@y1yang0
Copy link
Member Author

y1yang0 commented Sep 21, 2023

I had not noticed this PR sorry. I do not like the fact we have Linux-only functionality being added with no intent to supply similar functionality on other platforms. I also do not like the fact we had to ifdef the Linux code into the shared code. It would have been metter to place the code in os::Linux and then have a single LINUX_ONLY() to make that call.

Okay, I'll prepare a follow-up PR to do this once #15843 merged.

@y1yang0
Copy link
Member Author

y1yang0 commented Oct 11, 2023

I had not noticed this PR sorry. I do not like the fact we have Linux-only functionality being added with no intent to supply similar functionality on other platforms. I also do not like the fact we had to ifdef the Linux code into the shared code. It would have been metter to place the code in os::Linux and then have a single LINUX_ONLY() to make that call.

I made a simple attempt, but it is difficult to integrate this portion of code into OS Linux because it requires modifying the writer's written bytes and setting error messages. Logically, it belongs to DumpMerge. Abstracting it into a function similar to concatenate_file and placing it into OS Linux seems challenging.

@dholmes-ora
Copy link
Member

I think it can be improved somewhat even if we can't simply pass the _writer through.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hotspot-runtime hotspot-runtime-dev@openjdk.org integrated Pull request has been integrated serviceability serviceability-dev@openjdk.org
4 participants