Client connects forever with SSL/WSS

[bergice]

Describe what you would like to know or do

• Why does my client get stuck connecting forever to my WSS server?
• Is it possible to achieve this by setting up a DNS rule from my domain to route to the server IP, thus eliminating the need for coding the SSL support to the server directly?
• Do I need to do something on the client end other than connect via WSS to get this working? The client will be connecting via browser.

Describe the solution you'd considered

• Connecting without WSS works fine.

Additional context

• Server OS: Amazon Linux AMI, macOS Mojave
• Client OS: macOS Mojave
• JRE: 1.8
• WS Server: org.java-websocket:Java-WebSocket:1.4.0
• WS Client: com.github.czyzby:gdx-websocket-common:1.9.1.9.6

I'm thinking this has to do with the way I'm creating the SSL context.

The public/private pem files are retrieved directly from my registrar (no certbot).

Here is a replicateable example:

ServerExample.java

import org.java_websocket.WebSocket;
import org.java_websocket.handshake.ClientHandshake;
import org.java_websocket.server.DefaultSSLWebSocketServerFactory;
import org.java_websocket.server.WebSocketServer;

import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.SSLContext;
import javax.xml.bind.DatatypeConverter;
import java.io.ByteArrayInputStream;
import java.net.InetSocketAddress;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.security.KeyFactory;
import java.security.KeyStore;
import java.security.NoSuchAlgorithmException;
import java.security.cert.Certificate;
import java.security.cert.CertificateException;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.security.interfaces.RSAPrivateKey;
import java.security.spec.InvalidKeySpecException;
import java.security.spec.PKCS8EncodedKeySpec;

public class ServerExample {

	private static byte[] parseDERFromPEM( byte[] pem, String beginDelimiter, String endDelimiter ) {
		String data = new String( pem );
		String[] tokens = data.split( beginDelimiter );
		tokens = tokens[1].split( endDelimiter );
		return DatatypeConverter.parseBase64Binary( tokens[0] );
	}

	private static RSAPrivateKey generatePrivateKeyFromDER( byte[] keyBytes ) throws InvalidKeySpecException, NoSuchAlgorithmException {
		PKCS8EncodedKeySpec spec = new PKCS8EncodedKeySpec( keyBytes );

		KeyFactory factory = KeyFactory.getInstance( "RSA" );

		return ( RSAPrivateKey ) factory.generatePrivate( spec );
	}

	private static X509Certificate generateCertificateFromDER( byte[] certBytes ) throws CertificateException {
		CertificateFactory factory = CertificateFactory.getInstance( "X.509" );

		return ( X509Certificate ) factory.generateCertificate( new ByteArrayInputStream( certBytes ) );
	}

	public static SSLContext getContext(byte[] certBytes, byte[] keyBytes) {
		SSLContext context;
		char[] password = new char[0];

		try {
			// create certificates
			X509Certificate cert = generateCertificateFromDER( parseDERFromPEM(certBytes, "-----BEGIN CERTIFICATE-----", "-----END CERTIFICATE-----" ) );
			RSAPrivateKey key = generatePrivateKeyFromDER( parseDERFromPEM(keyBytes, "-----BEGIN PRIVATE KEY-----", "-----END PRIVATE KEY-----" ) );

			// create KeyStore
			KeyStore keystore = KeyStore.getInstance( "JKS" );
			keystore.load( null );
			keystore.setCertificateEntry( "cert-alias", cert );
			keystore.setKeyEntry( "key-alias", key, password, new Certificate[]{ cert } );

//            // create TrustManager
//            TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
//            tmf.init(keystore);

			// create KeyManager
			KeyManagerFactory kmf = KeyManagerFactory.getInstance( "SunX509" );
			kmf.init( keystore, password );

			// create the SSL context
			context = SSLContext.getInstance( "TLS" );
			context.init( kmf.getKeyManagers(), /*tmf.getTrustManagers()*/null, null );
		} catch ( Exception e ) {
			context = null;
		}

		return context;
	}

	public static void main (String[] strArgs) throws Exception {

		// Create the server
		WebSocketServer server = new WebSocketServer(new InetSocketAddress(6767)) {
			@Override
			public void onOpen(WebSocket conn, ClientHandshake handshake) {
				System.out.println("Connection opened " + conn.getRemoteSocketAddress().getHostName());
			}

			@Override
			public void onClose(WebSocket conn, int code, String reason, boolean remote) {
				System.out.println("Connection closed " + conn.getRemoteSocketAddress().getHostName());
			}

			@Override
			public void onMessage(WebSocket conn, String message) {
				System.out.println(conn.getRemoteSocketAddress().getHostName() + " says " + message);
			}

			@Override
			public void onError(WebSocket conn, Exception ex) {
				System.out.println("Error from " + conn.getRemoteSocketAddress().getHostName() + ": " + ex.getMessage());
			}

			@Override
			public void onStart() {
				System.out.println("Started");
			}
		};

		// Setup SSL - comment this section out to see this example working without WSS
		byte[] certBytes = Files.readAllBytes(Paths.get("cert.pem"));
		byte[] keyBytes = Files.readAllBytes(Paths.get("key.pem"));

		server.setWebSocketFactory( new DefaultSSLWebSocketServerFactory( getContext(certBytes, keyBytes) ) );

		// Set timeout
		server.setConnectionLostTimeout( 30 );

		// Set reuse address - to avoid `Address already in use`
		server.setReuseAddr(true);

		// Start the server
		server.start();
	}
}

ClientExample.java

import com.github.czyzby.websocket.CommonWebSockets;
import com.github.czyzby.websocket.WebSocket;
import com.github.czyzby.websocket.net.ExtendedNet;

public class ClientExample {

	public static void main (String[] arg) {
		CommonWebSockets.initiate();
//		WebSocket socket = ExtendedNet.getNet().newWebSocket("localhost", 6767); // this works
		WebSocket socket = ExtendedNet.getNet().newSecureWebSocket("localhost", 6767); // this does not work
		socket.connect();
		socket.send("Hello, world!");
		socket.close();
	}
}

[marci4]

Hello,

Please provide additional information (os and version, java and version, lib version) and an example code for both client and server...

Best regards,
Marcel

[bergice]

@marci4 no problem, I've added a condensed working example and more details.

[marci4]

Hello @bergice,

thank you for the additional info.
I have a few more question.

• Have you tried to connect with a browser to the server? What is the result there? (Maybe you need to trust your self signed certificate!)
• Check out Debugging SSL/TLS Connections

There was a bug on android which could be related to this issue. That would however indicate that your JRE has a bug.

Best regards,
Marcel

[bergice]

@marci4

No problem.

Have you tried to connect with a browser to the server? What is the result there? (Maybe you need to trust your self signed certificate!)

Yes, it just stalls:

Try it yourself if you'd like: https://lurkers.io/client/latest/index.html?host=18.213.159.248&nickname=Player&secure=true

Check out Debugging SSL/TLS Connections

Thanks, I looked through this and could not find anything that seemed relevant. I'm using valid SSL certificates from my registrar.

---

I'm wondering if you would be able to get the example above working?

I might try the SSL examples too and see if I can get them working.

[marci4]

Thanks, I looked through this and could not find anything that seemed relevant. I'm using valid SSL certificates from my registrar.

Well at least -Djavax.net.debug=all should be relevant since that would allow you do better understand the problem.

I'm wondering if you would be able to get the example above working?

I currently have no desktop pc where I can run and debug the example..

Best regards,
Marcel

[bergice]

I tried running both the server and the client with -Djavax.net.debug=all.

On the server it doesn't output any extra information but on the client it does. It just says that it's adding a bunch of certificates. I can send you the full log if you want.

[marci4]

I tried running both the server and the client with -Djavax.net.debug=all.

On the server it doesn't output any extra information but on the client it does. It just says that it's adding a bunch of certificates. I can send you the full log if you want.

No need, first of all it sounds like it is a SSL/configuration issue...

Cannot ping 18.213.159.248:6767 anyways... Firewall?

[bergice]

No need, first of all it sounds like it is a SSL/configuration issue...

I am going to try using JKS like in the examples as my code just uses the pem certificates, hopefully that works.

Cannot ping 18.213.159.248:6767 anyways... Firewall?

This is because the server has ICMP inbound traffic blocked.

[bergice]

I looked through my certificates and realised I was using the public key as the certificate. I changed it and now I am getting a ERR_CERT_COMMON_NAME_INVALID error.

[bergice]

I finally resolved this! 🎉

Here is what I had to do:

• Use the domain pem file provided by my registrar as my public key
• Setup a subdomain for my domain which routes to my server IP address, then connect through that
• Use CloudFront to serve the static website over HTTPS
• Fix DNS configuration for domain

Other observations worth noting:

• You can safely use just an SSLContext with just a private and public pem file (no need to use JKS)
• You can also connect from the client without an SSLContext as long as you route to the server through the domain using a CNAME DNS route

Thanks for the help @marci4

[marci4]

@bergice thanks your sharing your solution!