TLS-handshake fails and certificates not recognized

[PrivacyDragon]

When I run the server with the standard certificate stuff, the server successfully starts and is reachable.
Though the TLS-handshakes fail.

And when I try to configure vysper to use my certificate, it fails to start.

[PrivacyDragon]

This is the error with the standard settings

06:38:18,115 | WARN | org.apache.vysper.mina.XmppIoHandlerAdapter | error caught on transportation layer: {} javax.net.ssl.SSLHandshakeException: SSL handshake failed. at org.apache.mina.filter.ssl.SslFilter.messageReceived(SslFilter.java:495) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:119) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:693) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:646) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:635) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:67) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1079) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: javax.net.ssl.SSLException: No supported CertificateVerify signature algorithm for RSA key at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:358) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:305) at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:911) at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.onProduceCertificateVerify(CertificateVerify.java:1111) at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.produce(CertificateVerify.java:1104) at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:440) at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1257) at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1193) at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:851) at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:812) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) at org.apache.mina.filter.ssl.SslHandler.doTasks(SslHandler.java:767) at org.apache.mina.filter.ssl.SslHandler.handshake(SslHandler.java:541) at org.apache.mina.filter.ssl.SslHandler.messageReceived(SslHandler.java:349) at org.apache.mina.filter.ssl.SslFilter.messageReceived(SslFilter.java:476) ... 15 more

[PrivacyDragon]

With my own certificate :
Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'server' defined in class path resource [spring-config.xml]: Invocation of init method failed; nested exception is java.lang.RuntimeException: java.io.IOException: Invalid keystore format at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1420) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:580) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425) at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139) at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83) at org.apache.vysper.spring.ServerMain.main(ServerMain.java:38) Caused by: java.lang.RuntimeException: java.io.IOException: Invalid keystore format at org.apache.vysper.xmpp.server.DefaultServerRuntimeContext.setTlsContextFactory(DefaultServerRuntimeContext.java:214) at org.apache.vysper.xmpp.server.XMPPServer.start(XMPPServer.java:139) at org.apache.vysper.spring.SpringCompatibleXMPPServer.init(SpringCompatibleXMPPServer.java:69) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1544) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1485) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417) ... 12 more Caused by: java.io.IOException: Invalid keystore format at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:688) at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221) at java.base/java.security.KeyStore.load(KeyStore.java:1473) at org.apache.vysper.xmpp.cryptography.AbstractTLSContextFactory.createSSLContext(AbstractTLSContextFactory.java:84) at org.apache.vysper.xmpp.cryptography.AbstractTLSContextFactory.getSSLContext(AbstractTLSContextFactory.java:74) at org.apache.vysper.xmpp.server.DefaultServerRuntimeContext.setTlsContextFactory(DefaultServerRuntimeContext.java:212) ... 21 more

[PrivacyDragon]

I did something with keytools to create another, selfsigned, certificate that has a password and that kinda worked. I was able to get a client connected. But I had yo manually approve the certificate in the client or something. Still, sending messages does not work, because 'TLS-handshake times out'. I guess this is related to each other.

How can I just use the certificate that I use for all my webservers, which is a valid certificate?

There should really be more and better documentation.