Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create thread dumps for all running JVMs when the build gets cancelled #13100

Merged
merged 3 commits into from
Jan 9, 2023

Conversation

lhotari
Copy link
Contributor

@lhotari lhotari commented Jan 5, 2023

Motivation:

When a GitHub Actions build job gets cancelled because of a timeout, it would be useful to have a thread dump for all running JVM processes so that it would be able to find the reason why the build job timed out.

Modification:

Add an inline composite action that uses bash, jps and jcmd to create thread dumps for all running Java processes.
This is referenced in ci-pr.yml workflow's verify-pr job and gets run when the build job is cancelled.
This is an example of a thread dump that gets created: https://github.com/lhotari/netty/pull/1/checks#step:7:28 .

Copy link
Contributor

@chrisvest chrisvest left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool

.github/actions/thread-dump-jvms/action.yml Outdated Show resolved Hide resolved
@normanmaurer
Copy link
Member

@lhotari nice... do you also have a solution for windows / macOs ?

@lhotari
Copy link
Contributor Author

lhotari commented Jan 6, 2023

@lhotari nice... do you also have a solution for windows / macOs ?

@normanmaurer I'll work on that.

I also noticed that quite a few builds run the JVM within docker. I'll work on a solution that uses https://github.com/apangin/jattach for triggering the thread dumps since that tool works also for Java processes that run inside docker containers.

@lhotari lhotari requested a review from chrisvest January 9, 2023 09:02
@lhotari
Copy link
Contributor Author

lhotari commented Jan 9, 2023

@normanmaurer I have made changes to support java processes in docker containers as well as support for windows / macos.

Here's an example of the thread dump of a process running in a docker container: https://github.com/lhotari/netty/actions/runs/3872145028/jobs/6600659632#step:9:432

This example is from a windows container:
https://github.com/lhotari/netty/actions/runs/3872145028/jobs/6600659012#step:6:42

A small detail on windows is that the maven process gets killed when the GitHub Actions job gets cancelled and there will be no thread dump for the main maven process itself. On Linux, the maven main process isn't killed immediately and there will also be a thread dump available for that process. The thread dump for the surefire process will be available also on windows as seen in the above example.

@normanmaurer
Copy link
Member

@lhotari thanks so much! This is amazing

@normanmaurer normanmaurer added this to the 4.1.87.Final milestone Jan 9, 2023
@normanmaurer normanmaurer merged commit e19860a into netty:4.1 Jan 9, 2023
normanmaurer pushed a commit that referenced this pull request Jan 9, 2023
#13100)

Motivation:

When a GitHub Actions build job gets cancelled because of a timeout, it would be useful to have a thread dump for all running JVM processes so that it would be able to find the reason why the build job timed out.

Modification:

Add an inline composite action that uses bash, jps and jcmd to create thread dumps for all running Java processes. 
This is referenced in ci-pr.yml workflow's verify-pr job and gets run when the build job is cancelled.
This is an example of a thread dump that gets created: https://github.com/lhotari/netty/pull/1/checks#step:7:28 .

Result:

Easier to debug builds that did timeout.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants