8315702: jcmd Thread.dump_to_file slow with millions of virtual threads

[AlanBateman]

HotSpotDiagnosticMXBean.dumpThreads and jcmd Thread.dump_to_file are slow when there is a large number of threads.

The thread dump can be sped up significantly with some small changes:

• Using println rather than format when printing the thread info and thread stacks
• Wrap the underlying file output stream in a BufferedOutputStream

With 200k virtual threads on macOS, the plain format thread dump goes from 22s to 1.8s, and the json format thread dump goes from 31s to 2.8s on one system that I tried. On a Linux system, also with 200k threads, the plain thread dump goes from 8.7s to 2.9s, and the json format thread dump from 12.4s to 4.5s.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8315702: jcmd Thread.dump_to_file slow with millions of virtual threads (Bug - P4)

Reviewers

• Mandy Chung (@mlchung - Reviewer)
• Alex Menkov (@alexmenkov - Reviewer)
• Jaikiran Pai (@jaikiran - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/15581/head:pull/15581
$ git checkout pull/15581

Update a local copy of the PR:
$ git checkout pull/15581
$ git pull https://git.openjdk.org/jdk.git pull/15581/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 15581

View PR using the GUI difftool:
$ git pr show -t 15581

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/15581.diff

Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back alanb! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@AlanBateman The following label will be automatically applied to this pull request:

• core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[AlanBateman]

/label remove core-libs
/label add serviceability

[openjdk]

@AlanBateman
The core-libs label was successfully removed.

[openjdk]

@AlanBateman
The serviceability label was successfully added.

[mlbridge]

Webrevs

• 00: Full (42e41e40)

[schlosna]

I don't think more counts, but looks good to me.

[mlchung]

Looks good to me.

[openjdk]

@AlanBateman This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8315702: jcmd Thread.dump_to_file slow with millions of virtual threads

Reviewed-by: mchung, amenkov, jpai

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 17 new commits pushed to the master branch:

• 111ecdb: 8268829: Provide an optimized way to walk the stack with Class object only
• 716201c: 8314935: Shenandoah: Unable to throw OOME on back-to-back Full GCs
• 4c6d7fc: 8315795: runtime/Safepoint/TestAbortVMOnSafepointTimeout.java fails after JDK-8305507
• 7e7ab6e: 8315877: ProblemList vmTestbase/nsk/jvmti/InterruptThread/intrpthrd003/TestDescription.java on macosx-aarch64
• 0c865a7: 8315637: JDK-8314249 broke libgraal
• 683672c: 8292692: Move MethodCounters inline functions out of method.hpp
• 9bf3dee: 8314831: NMT tests ignore vm flags
• b74805d: 8315863: [GHA] Update checkout action to use v4
• 1cae0f5: 8315220: Event NativeLibraryLoad breaks invariant by taking a stacktrace when thread is in state _thread_in_native
• 8f7e29b: 8313422: test/langtools/tools/javac 144 test classes uses com.sun.tools.classfile library
• ... and 7 more: https://git.openjdk.org/jdk/compare/9887cd8adc408a71b045b1a4891cc0d5dede7e0e...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[jaikiran]

This looks good to me.

[AlanBateman]

Thanks for reviews.

[AlanBateman]

/integrate

[openjdk]

Going to push as commit 3c258ac.
Since your change was applied there have been 20 commits pushed to the master branch:

• 3a00ec8: 8312075: FileChooser.win32.newFolder is not updated when changing Locale
• 806ef08: 8315594: Open source few headless Swing misc tests
• 4b43c25: 8310929: Optimization for Integer.toString
• 111ecdb: 8268829: Provide an optimized way to walk the stack with Class object only
• 716201c: 8314935: Shenandoah: Unable to throw OOME on back-to-back Full GCs
• 4c6d7fc: 8315795: runtime/Safepoint/TestAbortVMOnSafepointTimeout.java fails after JDK-8305507
• 7e7ab6e: 8315877: ProblemList vmTestbase/nsk/jvmti/InterruptThread/intrpthrd003/TestDescription.java on macosx-aarch64
• 0c865a7: 8315637: JDK-8314249 broke libgraal
• 683672c: 8292692: Move MethodCounters inline functions out of method.hpp
• 9bf3dee: 8314831: NMT tests ignore vm flags
• ... and 10 more: https://git.openjdk.org/jdk/compare/9887cd8adc408a71b045b1a4891cc0d5dede7e0e...master

Your commit was automatically rebased without conflicts.

[openjdk]

@AlanBateman Pushed as commit 3c258ac.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

[robm-openjdk]

/backport jdk21u

[openjdk]

@robm-openjdk the backport was successfully created on the branch robm-openjdk-backport-3c258ac9 in my personal fork of openjdk/jdk21u. To create a pull request with this backport targeting openjdk/jdk21u:master, just click the following link:

➡️ Create pull request

The title of the pull request is automatically filled in correctly and below you find a suggestion for the pull request body:

Hi all,

This pull request contains a backport of commit 3c258ac9 from the openjdk/jdk repository.

The commit being backported was authored by Alan Bateman on 8 Sep 2023 and was reviewed by Mandy Chung, Alex Menkov and Jaikiran Pai.

Thanks!

If you need to update the source branch of the pull then run the following commands in a local clone of your personal fork of openjdk/jdk21u:

$ git fetch https://github.com/openjdk-bots/jdk21u.git robm-openjdk-backport-3c258ac9:robm-openjdk-backport-3c258ac9
$ git checkout robm-openjdk-backport-3c258ac9
# make changes
$ git add paths/to/changed/files
$ git commit --message 'Describe additional changes made'
$ git push https://github.com/openjdk-bots/jdk21u.git robm-openjdk-backport-3c258ac9