TLS Handshake failed error w/ OpenSSL 1.1.x

[vishnumotghare]

I'm trying to connect x11vnc server through VncViewer (TigerVnc). I'm giving following command to start vnc server with -ssl option

$ x11vnc -ssl -rfbauth /home/root/.vnc/passwd

When I'm trying to connect the vnc server, I'm getting TLS handshake failed error.
Following is the vncserver log

07/05/2020 13:25:58 SSL: accept_openssl(OPENSSL_VNC)
07/05/2020 13:25:58 SSL: spawning helper process to handle: 172.21.33.47:64181
07/05/2020 13:25:58 SSL: helper for peerport 64181 is pid 629:
07/05/2020 13:25:58 connect_tcp: trying: 127.0.0.1 20000
07/05/2020 13:25:59 check_vnc_tls_mode: waited: 1.422183 / 1.40 input: (future) RFB Handshake
07/05/2020 13:26:00 check_vnc_tls_mode: version: 3.8
07/05/2020 13:26:00 check_vnc_tls_mode: reply: 19 (VeNCrypt)
07/05/2020 13:26:00 vencrypt: received 0.2 client version.
07/05/2020 13:26:00 vencrypt: client selected sub-type: 257 (rfbVencryptTlsNone)
07/05/2020 13:26:00 Using Anonymous Diffie-Hellman mode.
07/05/2020 13:26:00 WARNING: Anonymous Diffie-Hellman uses encryption but is
07/05/2020 13:26:00 WARNING: susceptible to a Man-In-The-Middle attack.
07/05/2020 13:26:00 loaded Diffie Hellman 1024 bits, 0.001s
07/05/2020 13:26:00 SSL: ssl_init[629]: 6/6 initialization timeout: 20 secs.
07/05/2020 13:26:01 SSL: ssl_helper[629]: SSL_accept() *FATAL: -1 SSL FAILED
07/05/2020 13:26:01 SSL: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher
07/05/2020 13:26:01 SSL: ssl_helper[629]: Proto: unknown
07/05/2020 13:26:01 SSL: accept_openssl: cookie from ssl_helper[629] FAILED. 0

This was working properly without error. Only change is openssl version, It changed from OpenSSL 1.0.2p 14 Aug 2018 to OpenSSL 1.1.1d 10 Sep 2019.

What could be the reason for failure?

[mattcaswell]

07/05/2020 13:26:01 SSL: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher

The client and server have failed to find a common set of ciphersuites that they both support.

07/05/2020 13:26:00 Using Anonymous Diffie-Hellman mode.
07/05/2020 13:26:00 WARNING: Anonymous Diffie-Hellman uses encryption but is
07/05/2020 13:26:00 WARNING: susceptible to a Man-In-The-Middle attack.

This suggests you are using ADH based ciphersuites which are not available in the default security level in 1.1.1. This was not the case in 1.0.2, so my guess is that this is probably your problem.

I can demonstrate a similar problem using s_server/s_client in OpenSSL 1.1.1. Start s_server like this:

$ openssl s_server -cipher ADH-AES128-SHA -no_tls1_3

And s_client like this:

 openssl s_client -cipher "ADH-AES128-SHA:@SECLEVEL=0" -no_tls1_3

And you will see s_server complain with this error message:

140644863313152:error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher:ssl/statem/statem_srvr.c:2282:

Modifying the s_server command line args so that they look like this:

$ openssl s_server -cipher "ADH-AES128-SHA:@SECLEVEL=0" -no_tls1_3

And then retrying the s_client command should result in a successful connection. The difference is in the ciphersuite configuration where we explicitly specify a lower security level.

I am not familiar with VNC configuration, but you will need to do one of the following things:

1. Change the ciphersuite in use to not be based on ADH. This will require the installation of a server certificate.

2. Lower the security level in use to not complain about the use of ADH.

[vishnumotghare]

@mattcaswell Thanks for the help. I lowered the security level of ADH to ADH@SECLEVEL=0 in x11vnc server code & it worked.

[mattcaswell]

Great. Glad you got it working. Closing this.