Cannot find local keystore file from the classpath with embedded Tomcat 9.0.26 from executable jar

[phillipuniverse]

Looks like this is a regression caused by #18403 and the update to Tomcat 9.0.26. This was found on 2.1.9.RELEASE.

1. Generate a keystore file to be in the classpath
2. Configure Spring Boot to use SSL:

   server:
     port: 8443
      ssl:
        key-password: keypass
        key-store: classpath:local.keystore
        key-store-password: storepass

3. Build the jar with mvn package
4. Run the jar with java -jar target/demo.jar

This is the output:

2019-10-03 14:22:07.925 ERROR 6664 --- [           main] org.apache.catalina.util.LifecycleBase   : Failed to start component [Connector[HTTP/1.1-8443]]

org.apache.catalina.LifecycleException: Protocol handler start failed
	at org.apache.catalina.connector.Connector.startInternal(Connector.java:1008) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.apache.catalina.core.StandardService.addConnector(StandardService.java:227) [tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.addPreviouslyRemovedConnectors(TomcatWebServer.java:263) [spring-boot-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:195) [spring-boot-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.startWebServer(ServletWebServerApplicationContext.java:297) [spring-boot-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:163) [spring-boot-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:552) [spring-context-5.1.10.RELEASE.jar!/:5.1.10.RELEASE]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) [spring-boot-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744) [spring-boot-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391) [spring-boot-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:312) [spring-boot-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) [spring-boot-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1204) [spring-boot-2.1.9.RELEASE.jar!/:2.1.9.RELEASE]
	at com.example.demo.DemoApplication.main(DemoApplication.java:10) [classes!/:0.0.1-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_222]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_222]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [tomcat-ssl-bug-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [tomcat-ssl-bug-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) [tomcat-ssl-bug-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52) [tomcat-ssl-bug-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
Caused by: java.lang.IllegalArgumentException: Stream closed
	at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:99) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:218) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1124) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.apache.tomcat.util.net.AbstractEndpoint.start(AbstractEndpoint.java:1210) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.apache.coyote.AbstractProtocol.start(AbstractProtocol.java:585) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.apache.catalina.connector.Connector.startInternal(Connector.java:1005) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	... 22 common frames omitted
Caused by: java.io.IOException: Stream closed
	at java.util.zip.InflaterInputStream.ensureOpen(InflaterInputStream.java:67) ~[na:1.8.0_222]
	at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:142) ~[na:1.8.0_222]
	at org.springframework.boot.loader.jar.ZipInflaterInputStream.read(ZipInflaterInputStream.java:52) ~[tomcat-ssl-bug-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[na:1.8.0_222]
	at java.io.BufferedInputStream.read(BufferedInputStream.java:265) ~[na:1.8.0_222]
	at java.security.DigestInputStream.read(DigestInputStream.java:124) ~[na:1.8.0_222]
	at java.io.DataInputStream.readInt(DataInputStream.java:387) ~[na:1.8.0_222]
	at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:658) ~[na:1.8.0_222]
	at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) ~[na:1.8.0_222]
	at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) ~[na:1.8.0_222]
	at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) ~[na:1.8.0_222]
	at java.security.KeyStore.load(KeyStore.java:1445) ~[na:1.8.0_222]
	at org.apache.tomcat.util.security.KeyStoreUtil.load(KeyStoreUtil.java:69) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:217) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:206) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:283) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:247) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:97) ~[tomcat-embed-core-9.0.26.jar!/:9.0.26]
	... 28 common frames omitted

2019-10-03 14:22:07.929  INFO 6664 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2019-10-03 14:22:07.937  INFO 6664 --- [           main] ConditionEvaluationReportLoggingListener :

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2019-10-03 14:22:07.939 ERROR 6664 --- [           main] o.s.b.d.LoggingFailureAnalysisReporter   :

***************************
APPLICATION FAILED TO START
***************************

Description:

The Tomcat connector configured to listen on port 8443 failed to start. The port may already be in use or the connector may be misconfigured.

Action:

Verify the connector's configuration, identify and stop any process that's listening on port 8443, or configure this application to listen on another port.

Executing mvn spring-boot:run runs correctly and the server starts and is accessible.

Moving tomcat.version back to 9.0.24 within <properties> resolves the issue:

<properties>
    <tomcat.version>9.0.24</tomcat.version>
</properties>

Here is a gist with the offending pom.xml and application.yml

[wilkinsona]

Thanks for the report. Given that it works with Tomcat 9.0.24, I think this needs to be addressed in Tomcat rather than here. It sounds like a symptom of the problem described here that @mmoayyed brought to our attention on Gitter. It has already been fixed in this commit that will be in Tomcat 9.0.27. We'll upgrade to that in due course. In the meantime you can downgrade Tomcat as you have done or if you'd like to stick with 9.0.25 or .26, you could try using an SslStoreProvider bean to configure the keystore.

[PeterL1n]

Kinda disappointed having this kind of issue. I spent hours debugging on my own side. I feel recent Spring Boot releases have become much less stable.

[wilkinsona]

@PeterL1n sorry for the inconvenience. It's certainly unfortunate that we weren't aware of the regression in Tomcat before we released 2.1.9. We have integration tests for configuring Tomcat with SSL but the problem only occurs when Tomcat is loading the KeyStore from within a jar file rather than directly from the file system and the test doesn't cover that scenario.

Can you share some examples of other recent problems please? Perhaps we can then review them and identify an area where we could improve how we're doing things to stop them happening again.

[de-jcup]

For developers using the spring boot gradle plugin and having this issue, just add

ext['tomcat.version'] = '9.0.24'

to your build.gradle file and your have applied the former mentioned maven workaround for 2.1.9 in gradle style.

[mpriha]

I would like to comment that like said here the problem occurs also in a setup using file scheme e.g. server.ssl.key-store = file:/usr/local/myserver/keystore.pfx.

[phillipuniverse]

Confirmed fixed on 9.0.27 released last week. It just got deployed to Maven Central today, simply update tomcat.version to 9.0.27.