Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GHA: Archive JVM dumps #9129

Closed
Tracked by #9127
lenaschoenburg opened this issue Apr 13, 2022 · 2 comments · Fixed by #9223
Closed
Tracked by #9127

GHA: Archive JVM dumps #9129

lenaschoenburg opened this issue Apr 13, 2022 · 2 comments · Fixed by #9223
Assignees
@npepinpe
Labels
area/build Marks an issue as related to our build pipeline (e.g. Maven settings, CI, etc.) kind/toil Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc.

Comments

@lenaschoenburg
Copy link
Member

lenaschoenburg commented Apr 13, 2022 •
edited

Since we are loading and using native code with RocksDB, the JVM can segfault and then generate dumps that we should archive for investigations. In our Jenkins Pipeline this seems to be implemented here: https://github.com/camunda/zeebe/blob/7abb2c0b1dfd7f20181848aee365658cd6e0d327/Jenkinsfile#L325
@lenaschoenburg lenaschoenburg mentioned this issue Apr 13, 2022
Closed
19 tasks
@lenaschoenburg
Copy link
Member Author

@npepinpe The hs_err_*.log files are the one you were talking about, right?

@npepinpe
Copy link
Member

Yes, exactly. These are dumped when a segfault occurs, for example, and contain the stack trace so we can find what it was doing when it crashed. This hasn't happened in a while, but it can happen when playing with native memory/RocksDB.

@lenaschoenburg lenaschoenburg mentioned this issue Apr 13, 2022
Closed
@npepinpe npepinpe added kind/toil Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc. area/test Marks an issue as improving or extending the test coverage of the project labels Apr 14, 2022
@npepinpe npepinpe self-assigned this Apr 20, 2022
@npepinpe npepinpe mentioned this issue Apr 25, 2022
Merged
10 tasks
@npepinpe npepinpe added area/build Marks an issue as related to our build pipeline (e.g. Maven settings, CI, etc.) and removed area/test Marks an issue as improving or extending the test coverage of the project labels Apr 25, 2022
zeebe-bors-camunda bot added a commit that referenced this issue Apr 25, 2022
@zeebe-bors-camunda @npepinpe
merge: #9223
c4e095a 
9223: Archive JVM error dumps on failure r=npepinpe a=npepinpe

## Description

This PR adds an extra post step to archive JVM error dumps when the jobs have failed. This means archiving any file with the pattern `hs_err_*.log` produced by the job.

Archiving dump files generated by the JVM on crashes (e.g. SIGSEV, SIGBUS, etc.). This is particularly useful when dealing with native memory and running into unrecoverable crashes to get the stacktrace of the failing frame, as this is often not logged to standard out.

NOTE: the post steps are getting bigger, but there is a follow up issue to group these together.

## Related issues

closes #9129 



Co-authored-by: Nicolas Pepin-Perreault <nicolas.pepin-perreault@camunda.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@npepinpe npepinpe

Labels
area/build Marks an issue as related to our build pipeline (e.g. Maven settings, CI, etc.) kind/toil Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Archive JVM error dumps on failure camunda/camunda
2 participants
@npepinpe @lenaschoenburg