IOApp default contextshift implementation does not halt the JVM on fatal error

[kaarenilsen]

After getting a StackOverflowError I observed that my application (using IOApp) stopped working, but did not halt the JVM. I am running my app in kubernetes, and a JVM exit would result in the pod to be restarted, but since my IO stopped working, but the JVM still was running the result was basically that nothing was done in my app.

After discussing this in your gitter, receiving really good help we where able to boil this down to a simple example demonstrating the behavior, and a possible solution.

In my view I cannot see any good reason for the IOApp to continue to run after a fatal error, so I propose that the default ContextShift should exit the JVM in such situations.

The issue:

object ButWhy extends IOApp {
  override def run(args: List[String]): IO[ExitCode] = {
      val willFail = IO {
        if (true)
          throw new StackOverflowError("This will fail my code, skip error handler, and not exit the JVM")
        else
          throw new RuntimeException("This will fail the code and be handled by the error handler")
        "Success"
      }.handleErrorWith(t => IO("Failed"))

      Stream
        .eval(willFail)
        .to(_.evalMap(d => IO(println(d))))
        .repeat
        .compile
        .drain
        .as(ExitCode.Success)
  }
}

workaround:

object ButWhy extends IOApp {
  val log: Logger = LoggerFactory.getLogger(ButWhy.getClass)
  val mainEc: ExecutionContextExecutorService = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(2))

  def exitingEC(wrapped: ExecutionContext): ExecutionContext = new ExecutionContext {
    def execute(r: Runnable): Unit =
      wrapped.execute { () =>
        try r.run()
        catch {
          case t: Throwable =>
            log.error("Unhandled error occured.. Shutting down", t)
            System.exit(-1)
        }
      }
    def reportFailure(cause: Throwable): Unit =
      wrapped.reportFailure(cause)
  }

  override implicit protected def contextShift: ContextShift[IO] = IO.contextShift(exitingEC(mainEc))

  override def run(args: List[String]): IO[ExitCode] = {
    val willFail = IO {
      if (true)
        throw new StackOverflowError("This will fail my code, skip error handler, and not exit the JVM")
      else
        throw new RuntimeException("This will fail the code and be handled by the error handler")
      "Success"
    }.handleErrorWith(t => IO("Failed"))

    Stream
      .eval(willFail)
      .to(_.evalMap(d => IO(println(d))))
      .repeat
      .compile
      .drain
      .as(ExitCode.Success)
  }
}

[djspiewak]

Saw this in production. Didn't realize that ExecutionContext catches Throwable. This pretty much explains everything. Really nice catch!