Handshake failed on netty 4.1.21 running under Android 5.0

[MarkVilkel]

Hi!
I've got an exception on Android 5.0 with netty 4.1.21:

javax.net.ssl.SSLHandshakeException: Handshake failed
    io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Handshake failed
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) [na:0.0]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [na:0.0]
Caused by: javax.net.ssl.SSLHandshakeException: Handshake failed
	at com.android.org.conscrypt.OpenSSLEngineImpl.unwrap(OpenSSLEngineImpl.java:436) [na:0.0]
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:1006) [na:0.0]
Caused by: javax.net.ssl.SSLProtocolException: SSL handshake terminated: ssl=0x9d892e00: Failure in SSL library, usually a protocol error
                                                                             error:1408E0F4:SSL routines:SSL3_GET_MESSAGE:unexpected message (external/openssl/ssl/s3_both.c:498 0xac8b4ce0:0x00000000)
	at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake_bio(Native Method) [na:0.0]
    at com.android.org.conscrypt.OpenSSLEngineImpl.unwrap(OpenSSLEngineImpl.java:423) [na:0.0]

So Android app is unable to establish ssl connection.

I have had the same error when we were using mina library, however I've solved it by adding:

            if (!inNetBuffer.hasRemaining()) {
                res = new SSLEngineResult(SSLEngineResult.Status.BUFFER_UNDERFLOW, SSLEngineResult.HandshakeStatus.NEED_UNWRAP, 0, 0);

like it was suggested by Internets in org.apache.mina.filter.support.SSLHandler.unwrap0() method:

    private SSLEngineResult unwrap0() throws SSLException {
        SSLEngineResult res;
        do {
            if (SessionLog.isDebugEnabled(session)) {
                SessionLog.debug(session, "   inNetBuffer: " + inNetBuffer);
                SessionLog.debug(session, "   appBuffer: " + appBuffer);
            }
            if (!inNetBuffer.hasRemaining()) {
                res = new SSLEngineResult(SSLEngineResult.Status.BUFFER_UNDERFLOW, SSLEngineResult.HandshakeStatus.NEED_UNWRAP, 0, 0);
            } else {
                res = sslEngine.unwrap(inNetBuffer, appBuffer);
            }
            if (SessionLog.isDebugEnabled(session)) {
                SessionLog.debug(session, " Unwrap res:" + res);
            }
        } while (res.getStatus() == SSLEngineResult.Status.OK
                && (handshakeComplete && res.getHandshakeStatus() == SSLEngineResult.HandshakeStatus.NOT_HANDSHAKING
                        || res.getHandshakeStatus() == SSLEngineResult.HandshakeStatus.NEED_UNWRAP));
        
        return res;
    }

As I understand this is Android problem, described here https://issuetracker.google.com/issues/37017560
But we still have clients using Android 5.0, so we can not just ignore them.

Is not it possible to have such workaround in netty?

[normanmaurer]

@MarkVilkel yes I think so... let me check

[normanmaurer]

@MarkVilkel #7761 PTAL if this fixed it

[MarkVilkel]

@normanmaurer thank you for a quick fix.
We've tested d0b43d1 on Android 5.0 - works fine.
Waiting this fix in the next netty version.

[johnou]

@MarkVilkel do you see this problem with a Android 5.0 client talking SSL with a Netty server eg. standalone java app, or using Netty inside an Android application on 5.0?

[MarkVilkel]

@johnou we have an android application with netty client inside it connecting to netty transport server. Problem appeared only on Android 5.0