SSL: handshake failure (not support TLS1.2?)

[dontsovcmc]

I start to work with SSL for the first time and got error in MQTTClient_connect.

Environment:
Windows 10, x64, Visual Studio 12.0, openssl-1.0.2k.gz

Library compile script:

rmdir /s /q build.paho
mkdir build.paho
cd build.paho
call "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat" x64
cmake -G "Visual Studio 12 2013" -DPAHO_BUILD_STATIC=FALSE -DPAHO_WITH_SSL=TRUE -DOPENSSL_SEARCH_PATH="D:\CODE\adapters\var\env\win32_vs2012"  -DPAHO_BUILD_DOCUMENTATION=FALSE -DPAHO_BUILD_SAMPLES=TRUE -DCMAKE_CONFIGURATION_TYPES="Debug;Release" -DCMAKE_VERBOSE_MAKEFILE=TRUE .. 

I have CA file (ca.crt), Client Certificate File (client.crt), Client Key File (client.key) in PEM format.
I run broker with cafile (ca.crt), certfile (server.crt), keyfile (server.key)

I connect with MQTT.FX correctly using certificates, login, password.

When I connect by my application I see:

1508873022: New connection from 192.168.50.10 on port 8883.
1508873022: OpenSSL Error: error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure
1508873022: Socket error on client <unknown>, disconnectin

What I do:

MQTTClient_init_options opt = MQTTClient_init_options_initializer;
opt.do_openssl_init = true;
MQTTClient_global_init(&opt);

error_ = MQTTClient_create(&client,  "ssl://192.168.1.1:8883",  "XXXXX_id",
            MQTTCLIENT_PERSISTENCE_NONE, NULL);

MQTTClient_connectOptions conn_opts = MQTTClient_connectOptions_initializer;
 MQTTClient_SSLOptions ssl = MQTTClient_SSLOptions_initializer;

ssl.trustStore = "ca.crt";  //placed near exe file
ssl.keyStore = "client.crt";
ssl.privateKey = "client.key";
//ssl.enabledCipherSuites = "TLSv1.2";  //OpenSSL Error: error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure with and without this field
ssl.enableServerCertAuth = true;

conn_opts.ssl = &ssl;
conn_opts.keepAliveInterval = 20;
conn_opts.cleansession = true;
conn_opts.username = "user";
conn_opts.password = "pwd";
conn_opts.MQTTVersion = MQTTVERSION_DEFAULT;
conn_opts.connectTimeout = 10;

error_ = MQTTClient_setCallbacks(client, (void*)this, connection_lost, message_arrived, delivery_complete);
if (error_ == MQTTCLIENT_SUCCESS)
{
       error_ = MQTTClient_connect(client, &conn_opts);   //return -1 after ~7 sec
}

[dontsovcmc]

test3.exe output on my PC: https://pastebin.com/HvmNd1Wm

[icraggs]

As MQTT.fx is a Java client, the format of the truststore and keystore files is different from that used by OpenSSL. The test3.exe output indicates a likely problem with one of the files. You could try using the ones in this project to see what errors, if any, they give you.

[dontsovcmc]

Thanks a lot
I run test3.exe with paho example certificates... I can try with myself certs.

[dontsovcmc]

I create correct keys, thanks. But library doesnt' work :(

What I see in Wireshark:

First connection success by mosquitto_sub:
mosquitto_sub.exe -h 192.168.50.17 -p 8883 -d --insecure -t \sdgasd\g -v --cafile m2mqtt_ca.crt --cert client.crt --key client.key

Second failed by C code:

MQTTClient_create(&c, "ssl://192.168.50.17:8883", "multi_threaded_sample", MQTTCLIENT_PERSISTENCE_DEFAULT, NULL);

MQTTClient_SSLOptions ssl = MQTTClient_SSLOptions_initializer;

opts.ssl = &ssl;
opts.ssl->trustStore = "m2mqtt_ca.crt"; //options.server_key_file test-root-ca.crt",; /*file of certificates trusted by client*/
opts.ssl->keyStore = "client.crt"; //options.client_key_file client.pem;  /*file of certificate for client to present to server*/
opts.ssl->privateKey = "client.key"; 
//opts.ssl->enabledCipherSuites = "aNULL"; //"DEFAULT";  //
opts.ssl->enableServerCertAuth = 0;

opts.keepAliveInterval = 20;
opts.cleansession = 1;
opts.MQTTVersion = MQTTVERSION_DEFAULT;
opts.connectTimeout = 10;

rc = MQTTClient_setCallbacks(c, NULL, NULL, test2_messageArrived, test2_deliveryComplete);
assert("Good rc from setCallbacks", rc == MQTTCLIENT_SUCCESS, "rc was %d", rc);

std::cout << "Connecting";
rc = MQTTClient_connect(c, &opts);  // -1

[dontsovcmc]

I try to see connect of mosquitto_sub.exe under debugger and see that
socket opened before set up non-blocking

lib\net_mosq.c:442

In paho.mqtt.c: Socket.c socket set nonbloking before open. I got 10035 error after it :(.

[dontsovcmc]

A "good" news: async client failed same way :/.
I see in output

10468:error:20074002:BIO routines:FILE_CTRL:system lib:.\crypto\bio\bss_file.c:404:
10468:error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib:.\ssl\ssl_rsa.c:701:
10468:error:20074002:BIO routines:FILE_CTRL:system lib:.\crypto\bio\bss_file.c:404:
10468:error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib:.\ssl\ssl_rsa.c:701:
Connect failed, rc 0

The socket_error is here:

aClient->net->ctx is null ( SSLSocket_setSocketForSSL returns 0

[dontsovcmc]

I understand, that I set up TLS1.2 on the broker. Now I testing with TLS1

I change setup nonbloking socket after connect to socket like in Mosquitto library. Othewise I got 10035 error.

Now I debugging communicating

[dontsovcmc]

I connect successfully with setup nonbloking after connect function.. I can't understand, maybe Socket_setnonblocking doesn't work for Windows10x64. I try to find a reason on Monday.

Important note: I use full ways for certificates with \\ (double) in Windows.

[rashed043]

Hello dontsovcmc , I have almost similar problem, i cannot connect with SSL for publish the message. Could you please inform how did you configure or what is the program you have written so that it is working.

[dontsovcmc]

here is solution: #367
it merged with develope branch, i don't know is it in release ) i use my own patch

[dontsovcmc]

For TLS 1.2 - it isn't a SSL. TLS support here: #333

[rashed043]

thank you.. i am using vxworks. I have tried your provided solution. now i have a error on "SSLSocket_createContext". it is showing that "SSL_CTX_use_certificate_chain_file" has -1 error.
Could you please tell did you get with this kind of error?

[dontsovcmc]

Use google!
https://www.openssl.org/docs/man1.0.2/ssl/SSL_CTX_use_certificate_chain_file.html
SSL_CTX_use_certificate_chain_file() loads a certificate chain from file into ctx. The certificates must be in PEM format and must be sorted starting with the subject's certificate (actual client or server certificate), followed by intermediate CA certificates if applicable, and ending at the highest level (root) CA. There is no corresponding function working on a single SSL object.

[rashed043]

thank you..