Skip to content

Comments

GHA steps to collect and upload heap dumps to debug UT OOM errors#17029

Merged
abhishekrb19 merged 7 commits intoapache:masterfrom
abhishekrb19:collect_heap_dumps_ut_oom
Sep 12, 2024
Merged

GHA steps to collect and upload heap dumps to debug UT OOM errors#17029
abhishekrb19 merged 7 commits intoapache:masterfrom
abhishekrb19:collect_heap_dumps_ut_oom

Conversation

@abhishekrb19
Copy link
Contributor

@abhishekrb19 abhishekrb19 commented Sep 10, 2024

Some UTs have recently been failing with OOMs: https://github.com/apache/druid/actions/runs/10796374752/job/29955096914?pr=17027

Warning:  Tests run: 15, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.868 s -- in org.apache.druid.msq.test.CalciteUnionQueryMSQTest
[INFO] Running org.apache.druid.msq.test.CalciteSelectQueryMSQTest
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid21084.hprof ...
Heap dump file created [3035323731 bytes in 8.410 secs]
Terminating due to java.lang.OutOfMemoryError: Java heap space

So this PR adds some GHA steps to tarball and upload the heap dump if they exist.

This PR has:

  • been self-reviewed.

Co-authored-by: Elliott Freis <108356317+imply-elliott@users.noreply.github.com>
@abhishekrb19 abhishekrb19 force-pushed the collect_heap_dumps_ut_oom branch from 6923b17 to 2920c2e Compare September 11, 2024 16:23
Copy link
Member

@kgyrtkirk kgyrtkirk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

project.parent.basedir is already used ; and the module hierarchy is usually 1 level...we could accept the same risks...
+1

@abhishekrb19 abhishekrb19 merged commit c077daa into apache:master Sep 12, 2024
@abhishekrb19 abhishekrb19 deleted the collect_heap_dumps_ut_oom branch September 12, 2024 13:06
cecemei pushed a commit to cecemei/druid that referenced this pull request Sep 12, 2024
…ache#17029)

* Add GHA steps to tar and upload any heap dumps on failure to debug UT OOM issues.

* Add jvm options to heap dump OnOutOfMemoryError

Co-authored-by: Elliott Freis <108356317+imply-elliott@users.noreply.github.com>

---------

Co-authored-by: Elliott Freis <108356317+imply-elliott@users.noreply.github.com>
abhishekrb19 added a commit that referenced this pull request Sep 13, 2024
#17054)

A command line arg -XX:OnOutOfMemoryError='chmod 644 ${project.parent.basedir}/target/*.hprof' was added to collect heap dumps: #17029

This arg is causing problems when running tests from Intellij. Intellij doesn't seem to likechmod 644, but this command works as expected in mvn. So as a workaround, add the chmod 644 ${BASE_DIR/target/*.hprof' command in a shell script that can then be executed when OnOutOfMemoryError happens to make Intellij happy.
pranavbhole pushed a commit to pranavbhole/druid that referenced this pull request Sep 17, 2024
apache#17054)

A command line arg -XX:OnOutOfMemoryError='chmod 644 ${project.parent.basedir}/target/*.hprof' was added to collect heap dumps: apache#17029

This arg is causing problems when running tests from Intellij. Intellij doesn't seem to likechmod 644, but this command works as expected in mvn. So as a workaround, add the chmod 644 ${BASE_DIR/target/*.hprof' command in a shell script that can then be executed when OnOutOfMemoryError happens to make Intellij happy.
@adarshsanjeev adarshsanjeev added this to the 32.0.0 milestone Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants