JVM can't exit due to threads left if Tomcat throws exceptions during shutdown

[jpmsilva]

Summary
If TomcatWebServer#rethrowDeferredStartupExceptions throws an exception, the embedded Tomcat instance may not allow the the JVM to terminate.

Symptoms
We have been running into some sporadic issues in which startup errors will sometime leave the JVM running, although the main thread has since died (the one running the @SpringBootApplication main).

Things like a failing Liquibase migration, that were being applied fairly early during the startup of the application, would leave the service hanging, and not truly failing (exiting with a status code != 0).

Analysis
The problem stems from the fact that some beans require early initialization in order for them to be injected into the Tomcat context.
Beans like Servlet Filters (and Servlet Filter Registrations) or Servlet Listeners (and Servlet Listener Registrations) are needed before the Tomcat context is started, so they (and their dependents) are created very early.
The remaining beans are created after the Tomcat context has been created.

If an exception occurs after the Tomcat context is started, the Tomcat is properly disposed of. However, if an error occurs early, the instance will not be fully created, and the instance will not be closed and destroyed.

This leaves some non daemon threads running and, those kind of threads do not allow the JVM to terminate, even if the exception propagates all the way to the main method.

Technical background
The method org.springframework.boot.web.embedded.tomcat.TomcatStarter#onStartup is called to run all ServletContextInitializers
Spring Boot uses a ServletContextInitializer to initialize the Spring framework that comes from org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext#getSelfInitializer.

This ServletContextInitializer calls org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext#getServletContextInitializerBeans to fetch available org.springframework.boot.web.servlet.ServletContextInitializerBeans

• Beans like Servlet Filters and Servlet Listeners are initialized here
• If such beans throw errors (or one of their dependents) the exception is deferred and stored in the catch clause of org.springframework.boot.web.embedded.tomcat.TomcatStarter#onStartup

Back to org.springframework.boot.web.embedded.tomcat.TomcatWebServer#initialize, after tomcat.start() is done, org.springframework.boot.web.embedded.tomcat.TomcatWebServer#rethrowDeferredStartupExceptions is called to rethrow any deferred exceptions, which is immediately caught and:

• the Tomcat server is stopped silently with org.springframework.boot.web.embedded.tomcat.TomcatWebServer#stopSilently
• checked Exception is wrapped in a WebServerException and
• result is thrown

Since initialize() is called from the constructor of TomcatWebServer, a webServer instance is never stored in org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext#createWebServer

Down the stack, org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext#refresh will catch the WebServerException and call org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext#stopAndReleaseWebServer.
However, since a webServer was never fully initialized, there is none to call stop() upon.

Notice that stop() is not the same as stopSilently(). stop() will stop and destroy and destroy the Tomcat instance. stopSilently() does not destroy the Tomcat instance.

Root cause
A Tomcat server (org.apache.catalina.Server) depends on destroy to release certain resources.
One such resource is the utilityExecutor stored in org.apache.catalina.core.StandardServer.
This executor creates threads that are not marked as daemon (org.apache.catalina.core.StandardServer#utilityThreadsAsDaemon is by default false).
Since these threads are not daemon, they do not allow the JVM to die after the main thread finishes.
The executor is stopped from org.apache.catalina.core.StandardServer#destroyInternal, which depends on Tomcat being destroyed.

[pivotal-issuemaster]

@jpmsilva Please sign the Contributor License Agreement!

Click here to manually synchronize the status of this Pull Request.

See the FAQ for frequently asked questions.

[pivotal-issuemaster]

@jpmsilva Thank you for signing the Contributor License Agreement!

[jpmsilva]

I apologize beforehand for the denseness of the technical analysis, but I wanted to leaved well documented the rationale behind the fix, since the issue may be complicated to grasp, given that certain bean creation failures will trigger bug, while others will not.

[philwebb]

apologize beforehand for the denseness of the technical analysis

@jpmsilva there's no need to apologize, the deep analysis is extremely useful!

[philwebb]

@jpmsilva Thanks for the PR. This has now been merged to 2.1.x and master