Revisit logging error stack traces in cases of fatal errors

[jrudolph]

Right now the error and stack trace is only logged on stderr if akka.jvm-exit-on-fatal-error is enabled. It's not clear if that was intended or an oversight.

Context: https://groups.google.com/d/topic/akka-user/R9utH4VIeXU/discussion

@raboof:

Is the logging really logged to stderr here though? Looking at the code:

        if (settings.JvmExitOnFatalError) {
          try {
            markerLogging.error(LogMarker.Security, cause, "Uncaught error from thread [{}] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled", thread.getName)
            import System.err
            err.print("Uncaught error from thread [")
            err.print(thread.getName)
            err.print("] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[")
            err.print(name)
            err.println("]")
            cause.printStackTrace(System.err)
            System.err.flush()
          } finally {
            System.exit(-1)
          }
        } else {
          markerLogging.error(LogMarker.Security, cause, "Uncaught fatal error from thread [{}] shutting down ActorSystem [{}]", thread.getName, name)
          terminate()
        }

'Uncaught fatal error' is in the 'else' block and I don't see a "cause.printStackTrace" there.

Do we have a reason for ignoring the 'cause' in MarkerLoggingAdapter?

@jrudolph:

I think the general idea is to do as little as possible on fatal errors because simple code is more likely to "make it through" (e.g. to prevent logging code provoking another OOM). This is a pretty vague rationale but fatal errors could have lots of causes so we can only guess and try to be conservative.

That said, I don't think there's a good reason to structure the code like this. I think a better solution might be to log the error to stderr first (in all cases) and then also log to the logger afterwards. Using the logger is more likely to run into extra problems as it might need the network still to work etc.

[raboof]

Created #23117. Included the other system termination logging changes too because adding the stacktrace to the logging might make it heavier, so logging to stderr first to be on the safe side.