How to turn on hs_err_* reporting on JVM crashes on MacOS?

[dolda2000]

Question

I've been trying to track down a crash issue that happens on MacOS, but I can't seem to figure out how to make the JVM write hs_err* log files like it does on Windows and Linux. Impractically enough, I don't have any Apple hardware myself, so I find myself only in temporary access at friends and in stores, making it difficult to do any in-depth research at the time.

What happens is that MacOS seems to do its own error reporting, which gives its own log-file, but since MacOS can't decode Java stack frames, this log file isn't nearly as useful as hs_err_* logs have been on other platforms. Is deferring to MacOS' crash handler a deliberate choice, or is it something that is just difficult to avoid for some reason? In the former case, I'd expect there would be some sort of command-line option to re-enable hs_err_* logging, but I have not been able to find it.

Context

Java version: Temurin-17.0.7+7 (build 17.0.7+7)

Your operating system and platform: Darwin 21.1.0 Kernel Version 21.1.0 root:xnu-8019.41.0-1/RELEASE_ARM64_T8101 arm64

[karianna]

By default, the JVM should create this file in the working directory of the process at the time of the crash (or $TEMP). For instance, if you launched your application from the terminal in your home directory, that's where the crash log should be created.

However, if the JVM is not creating this file, it may be because it doesn't have write permissions to the directory. You can specify the location where the JVM should create the hs_err_pid file by using the java command-line option -XX:ErrorFile. e.g.,

java -XX:ErrorFile=/path/to/log/directory/hs_err_pid%p.log -jar your_program.jar

[dolda2000]

By default, the JVM should create this file in the working directory of the process at the time of the crash (or $TEMP). For instance, if you launched your application from the terminal in your home directory, that's where the crash log should be created.

That is what I would have thought, too, but that is not what happens. The home directory is in fact where I'm running Java from, but there's no hs_err_* file there after the process crashes. I've even tried running the program from another directory where I know the JVM has write access (because it successfully writes other files to it before it crashes) just in case MacOS has some kind of complex per-program permissions or something on the home directory, but still nothing.

I wasn't aware of the -XX:ErrorFile option, so I tried using that explicitly now, but the JVM still doesn't create the file I specify with it. I tried giving both absolute and relative paths just in case, but neither worked.

It is perhaps worth noting that my crash problem seems to be specific to Apple Silicon, so that's the kind of hardware that I'm testing on, in case that makes a difference.

[dolda2000]

By default, the JVM should create this file in the working directory of the process at the time of the crash (or $TEMP).

In fact, if that really is how it's supposed to work, but not actually how it does work, does that mean I should open a bug report instead of a support request?

[dolda2000]

In fact I've searched my whole home directory (with find), and there are no hs_err_* files at all, so I'd like to think that I can at least exclude the possibility that I'm confusing the directory it's supposed to end up in.

[github-actions]

We are marking this issue as stale because it has not been updated for a while. This is just a way to keep the support issues queue manageable.
It will be closed soon unless the stale label is removed by a committer, or a new comment is made.

[karianna]

@dolda2000 sorry to let this one slip. What version of Mac OS X are you running and what was the location you specified. Was the Java process running as the same user as that location, i.e., would the Java process have permission to write to that location.

[dolda2000]

Hi! The version of Mac OS X should be stated in the OP. I don't know exactly what name of OSX v21 corresponds to, but Wikipedia seems to think it's Monterey.
I don't remember the exact location where I tried to write the hs_err file, but it was the same directory that I had downloaded Java and my Jar file to (and it would have been somewhere in the user's home directory), so unless OSX has some sort of per-program permissions that I'm not aware of, it should have been quite writable by the user in question. In that context, it might be worth mentioning that the Java program was capable of writing files to various places in the user's home directory before crashing, at least.

Is the JVM supposed to also invoke Mac OS X's own error reporting facility, or is it just taking over for some reason?

[karianna]

Do you get the crash message from the customer/user on stdout/stderr? It usually has a line saying something like trying to write hs_err_* to a location and can give a hint as to why that's not occuring.

[dolda2000]

Do you get the crash message from the customer/user on stdout/stderr?

It's coming from myself, it's just that I don't have permanent access to any Mac hardware, so I'm sneaking into stores and stuff to gain temporary access

It usually has a line saying something like trying to write hs_err_* to a location and can give a hint as to why that's not occuring.

It's been a while since I last tried now, but I'm pretty sure it was just silent. Can probably try again within the not-too-distant future if need be, but I'm fairly certain. The way it looks to me, it's just like OSX's crash reporting mechanism was triggering instead of the JVM's own.

Would the actual crash report from OSX help in debugging this? I'm attaching it just in case. Perhaps it is something to do with the signal being SIGTRAP rather than SIGABRT or SIGSEGV?
crash.txt

[karianna]

Hmm, that is odd, are you installing Java via the tar.gz archive or a DMG/PKG?

Looks like you might be using LWJGL - is this this Minecraft by any chance?

[dolda2000]

I'm using the tarball, and no, it's not Minecraft, it's Haven & Hearth.

[karianna]

I'd try using the package installer and see if that makes a difference (it should add in the correct hooks).

[dolda2000]

I'd try using the package installer and see if that makes a difference
I would, but as I said, I only have access to Mac hardware on demo computers in stores, basically, and I don't know the account password to do a package installation.

However, I also just received an error report from a user which looks extremely similar, but indicates a JDK installation path of /Library/Java/JavaVirtualMachines/jdk-21.jdk/Contents/Home/bin/java, which I would assume means that he used the package installer.

Is it normal for the JDK to dump both the system error report and the hs_err_* report, or should I assume whenever the system error report gets dumped that there is no hs_err_* report to be found?

[karianna]

I'll try to give a fuller picture and see if this helps with the diagnosis (it may be a type of crash that doesn't create a log, see below).

When Java crashes on macOS, it often produces an hs_err_pid.log file, which is a detailed crash log specifically for the Java Virtual Machine (JVM). This file contains diagnostic information about the JVM state at the time of the crash, such as Java thread stacks, native stack traces, heap information, and more. This file is invaluable when diagnosing issues with the JVM itself or with Java applications that result in JVM-level faults.

However, whether a macOS crash log (typically found in the "Console" application under "User Diagnostic Reports" or "System Diagnostic Reports") is generated in addition to the hs_err_pid.log file depends on the nature of the crash:

1. JVM Internal Crash: If the JVM itself encounters an internal error or fault (e.g., a segmentation fault or a fatal error in the JVM), then it will generate an hs_err_pid.log. In such cases, macOS might also generate its own crash report because a native process on the system has crashed.

2. Java Application-Level Exception: If the crash is due to an unhandled Java exception (e.g., a NullPointerException or ArrayIndexOutOfBoundsException), then only the Java application will be aware of the crash, and it won't result in a JVM crash. Thus, you will not see an hs_err_pid.log, and macOS won't generate a crash log either.

3. Native Code Crash: If your Java application uses JNI (Java Native Interface) or some other form of native integration, and the crash happens in the native code, then both an hs_err_pid.log and a macOS crash report may be generated.

4. Other System-level Interruptions: There might be situations where the JVM is killed due to external factors like system overloads, out-of-memory situations, or manual termination. In such cases, you might not get an hs_err_pid.log, but macOS might generate its own logs or reports about the event.

In summary, while the presence of an hs_err_pid.log file usually indicates a JVM crash, the presence or absence of a macOS crash log depends on the nature and context of the crash. However, it's possible for both logs to be generated for certain types of crashes. If you're debugging a crash, it's a good idea to check for both logs as they can provide complementary information.

[dolda2000]

As far as I can tell from the MacOS crash log, this crash seems to be what you classify as "Native Code Crash", as it seems to be crashing in AppKit:

Thread 47 Crashed:: Java: Render thread
0   AppKit                                     0x19e186118 -[NSOpenGLContext setView:] + 212
1   AppKit                                     0x19e4e1b84 -[_NSOpenGLViewBackingLayer display] + 468
2   QuartzCore                                 0x1a21b63dc CA::Layer::display_if_needed(CA::Transaction*) + 760
3   QuartzCore                                 0x1a2329b9c CA::Context::commit_transaction(CA::Transaction*, double, double*) + 456
4   QuartzCore                                 0x1a2198be8 CA::Transaction::commit() + 648
5   ???                                        0x115d198ac ???
6   ???                                        0x115d15ec0 ???
...

Any thoughts on my previous speculation that it might be because the JVM receives SIGTRAP rather than something more normal like SIGSEGV or SIGABRT? Could it be that SIGTRAP is something the JVM doesn't catch, and therefore doesn't leave a hs_err_pid.log?

[karianna]

The JVM does have mechanisms to handle various signals, including SIGTRAP, especially in the context of macOS. Here are some details regarding SIGTRAP and the JVM:

1. HotSpot and Signals: The Java HotSpot Virtual Machine, which is the most commonly used JVM, employs signal handlers to deal with exceptional situations. For example, it uses signals to implement features like safepoints, garbage collection pauses, and explicit null checks, among others.

2. SIGTRAP and macOS: On macOS, SIGTRAP is notably used by the system to indicate an assertion failure or other similar conditions. If a native method (perhaps invoked via JNI) in a Java program triggers a SIGTRAP, the JVM's default behavior would be to handle it, print diagnostic information (like the hs_err_pid.log), and terminate the process.

3. Overriding JVM Signal Handlers: By default, the JVM will install its own signal handlers for the signals it's interested in. However, you can use the JVM option -Xrs (Reduce Signal Usage) to prevent the JVM from handling certain signals, which can be helpful if you have a native library that needs to handle those signals. But note that using -Xrs can also make the JVM less resilient to certain error scenarios because it won't be able to capture and log the diagnostic information related to those signals.

4. HotSpot's Use of SIGTRAP: In some JVM internal implementations, SIGTRAP might be used to handle certain events, such as implicit null checks. When a Java program tries to access a null object, rather than explicitly checking for null, the JVM might rely on the hardware to throw an access violation exception. This exception is then caught by the JVM's signal handler, and if it's due to a null object access, the JVM translates it to a NullPointerException.

In conclusion, while the JVM does handle SIGTRAP signals, the exact behavior might vary based on the context and the specific JVM version or configuration. If you're dealing with native code and encountering issues related to signal handling, it's crucial to consider the JVM's interaction with signals and potentially adjust configurations as needed.

[github-actions]

We are marking this issue as stale because it has not been updated for a while. This is just a way to keep the support issues queue manageable.
It will be closed soon unless the stale label is removed by a committer, or a new comment is made.

[dolda2000]

Just for the record, this is still an issue. I just don't know how to debug it.

[Redgreenblack]

I have been try to find a solution to the issue Dolda mentioned for a while too. Can we keep this issue open for the sake of raising visibility on the issue? @karianna

[dolda2000]

I see this is tagged waiting on me, but I'm not sure what it's waiting for. Is there anything I can do to further it?

[karianna]

I removed that tag :-)

[github-actions]

We are marking this issue as stale because it has not been updated for a while. This is just a way to keep the support issues queue manageable.
It will be closed soon unless the stale label is removed by a committer, or a new comment is made.