Error in Open SSL Certificate

[amiya068]

Hi,

I have written an application for connecting to AzureIOT hub using AMQP protocol.
When i run the application it fails because of SSL issue.

Any help would be appreciate..
Below are the details for the OS
Yocto linux
Kernel 4.4.19-gdb0b54cdad

Info: IoT Hub SDK for C, version 1.1.19

Info: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
Error: Time:Tue Aug 8 13:57:57 2017 File:/home/ifrohner/HVAC/IOT_latest/azure-iot-sdk-c/c-utility/src/wsio.c Func:internal_close Line:150 wsio_close when not open.
Error: Time:Tue Aug 8 13:57:57 2017 File:/home/ifrohner/HVAC/IOT_latest/azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_common.c Func:on_amqp_connection_state_changed Line:635 amqp_connection was closed unexpectedly; connection retry will be triggered.
Info: Transport state changed from AMQP_TRANSPORT_STATE_CONNECTED to AMQP_TRANSPORT_STATE_RECONNECTION_REQUIRED
Error: Time:Tue Aug 8 13:57:57 2017 File:/home/ifrohner/HVAC/IOT_latest/azure-iot-sdk-c/uamqp/src/amqp_management.c Func:amqp_management_close Line:888 AMQP management instance not open
Error: Time:Tue Aug 8 13:57:57 2017 File:/home/ifrohner/HVAC/IOT_latest/azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_connection.c Func:on_cbs_open_complete Line:149 CBS open failed
Info: Preparing transport for re-connection
Error: Time:Tue Aug 8 13:57:57 2017 File:/home/ifrohner/HVAC/IOT_latest/azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_cbs_auth.c Func:on_cbs_put_token_complete_callback Line:188 CBS reported status code 0, error: '(null)' for put-token operation for device '8ccaee1b-d70b-44b4-a5ad-9cc8f93f9993'
Error: Time:Tue Aug 8 13:57:57 2017 File:/home/ifrohner/HVAC/IOT_latest/azure-iot-sdk-c/uamqp/src/saslclientio.c Func:saslclientio_send Line:1133 send called while not open
Error: Time:Tue Aug 8 13:57:57 2017 File:/home/ifrohner/HVAC/IOT_latest/azure-iot-sdk-c/uamqp/src/connection.c Func:on_bytes_encoded Line:241 Cannot send encoded bytes
Error: Time:Tue Aug 8 13:57:57 2017 File:/home/ifrohner/HVAC/IOT_latest/azure-iot-sdk-c/uamqp/src/saslclientio.c Func:saslclientio_close Line:1083 saslclientio_close called while not open
Error: Time:Tue Aug 8 13:57:57 2017 File:/home/ifrohner/HVAC/IOT_latest/azure-iot-sdk-c/uamqp/src/connection.c Func:on_bytes_encoded Line:245 xio_close failed
Error: Time:Tue Aug 8 13:57:57 2017 File:/home/ifrohner/HVAC/IOT_latest/azure-iot-sdk-c/uamqp/src/saslclientio.c Func:saslclientio_close Line:1083 saslclientio_close called while not open
Error: Time:Tue Aug 8 13:57:57 2017 File:/home/ifrohner/HVAC/IOT_latest/azure-iot-sdk-c/uamqp/src/connection.c Func:connection_close Line:1339 xio_close failed
Info: Transport state changed from AMQP_TRANSPORT_STATE_RECONNECTION_REQUIRED to AMQP_TRANSPORT_STATE_READY_FOR_RECONNECTION
Info: Transport state changed from AMQP_TRANSPORT_STATE_READY_FOR_RECONNECTION to AMQP_TRANSPORT_STATE_RECONNECTING
Info: Transport state changed from AMQP_TRANSPORT_STATE_RECONNECTING to AMQP_TRANSPORT_STATE_CONNECTED
The specified SAS token has an invalid signature. It does not match either the primary or secondary key of the device
Error: Time:Tue Aug 8 13:57:57 2017 File:/home/ifrohner/HVAC/IOT_latest/azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_common.c Func:IoTHubTransport_AMQP_Common_Device_DoWork Line:996 Failed performing DoWork for device '8ccaee1b-d70b-44b4-a5ad-9cc8f93f9993' (device reported state 4; number of previous failures: 0)
Info: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed

i am not sure why this issue is appearing, it looks like an openssl issue. But i do have the openssl certificates in the below location,
"/etc/ssl/certs/ca-certificates.crt"

[markrad]

In the past I've seen OpenSSL on Yocto builds ignore the root certificate location. Try passing the certificate via the IoTHubClient_LL_SetOption with "TrustedCerts" option value. You can find the certificate in certs.c. You only need to use the first certificate in the array. If you have the OpenSSL command on your device you can try openssl version -d. This will tell you where it is looking for certificates. The openssl s_client command can also be a useful debugging tool.

Mark Radbourne MSFT

[czyba]

Hi @amiya068,

There may also be another issue. If you are using openssl you most likely require certain "hashed" symlinks in order for openssl to find your certificate. They are normally installed in "post-install" steps for most distributions (i.e. debian / fedora ) when you install the respective package. I believe most distributions do this by running a perl script, but I haven't checked in a while.

I assume you have an embedded system. This may require you to calculate the symlinks beforehand. You can find an explanation here in section Installing CA Certificates into the OpenSSL framework. You would need to create the symlinks either on the device when it boots for the first time or you could just make them part of the device image.

Note: Installing the certificate will make it available for every program. If you do not wish to do that, follow @markrad's advice and use the TrustedCerts option. This will make the certificate only available for the IoT application instead of the whole system.

If you wish to see what it should look like, check out ls /etc/ssl/certs in a desktop linux of your choice. They will most likely contain the symlinks for all certificates in the mozilla certificate collection, unless you take a look at something more exotic.

I hope this helps. Kind Regards,
Christopher Czyba

[amiya068]

Hi @markrad,

It is using the "TrustedCerts" option with IoTHubClient_LL_SetOption.
But executing the command openssl version -d it displayed the following,

-sh-3.2# openssl version -d
OPENSSLDIR: "/usr/lib/ssl"

But the actual certificates are located under /etc/ssl/ folder, so i copied all the certificates under /usr/lib/ssl folder but still there was no luck with this.
OPENSSL version 1.0.2h is currently installed.

When i executed the command openssl s_client -connect cbpt01atqav2.azure-devices.net:443 it displayed the following,

-sh-3.2# openssl s_client -connect cbpt01atqav2.azure-devices.net:443
CONNECTED(00000004)
depth=1 C = US, ST = Washington, L = Redmond, O = Microsoft Corporation, OU = Microsoft IT, CN = Microsoft IT SSL SHA2
verify error:num=20:unable to get local issuer certificate

Certificate chain
0 s:/CN=*.azure-devices.net
i:/C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/OU=Microsoft IT/CN=Microsoft IT SSL SHA2
1 s:/C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/OU=Microsoft IT/CN=Microsoft IT SSL SHA2
i:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root

Server certificate
-----BEGIN CERTIFICATE-----
(Server certificate is displayed properly here....)
-----END CERTIFICATE-----
subject=/CN=*.azure-devices.net
issuer=/C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/OU=Microsoft IT/CN=Microsoft IT SSL SHA2

No client certificate CA names sent
Peer signing digest: SHA1
Server Temp Key: ECDH, P-256, 256 bits

SSL handshake has read 3662 bytes and written 473 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-SHA256
Session-ID: 02270000B2EA76D2ED4ED76E278B72F61FDF239EF52E7BE64F245803ACDF4A19
Session-ID-ctx:
Master-Key: A5CD728B4EFBB2B883D0252613B450647C19B780C036B07CD7B4B24F6202218A7502AD0AF086BBB3D095D7EB819B30F9
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1502288126
Timeout : 300 (sec)
Verify return code: 20 (unable to get local issuer certificate)

Not sure why the last line says "unable to get local issuer certificate".

Also few more debugging info is displayed below,

-sh-3.2# openssl s_client -verify arg
verify depth is 0
connect: Connection refused
connect:errno=111
-sh-3.2# openssl s_client -verify_return_error
connect: Connection refused
connect:errno=111
-sh-3.2# openssl s_client -cert /etc/ssl/certs/client-cert.pem
unable to load client certificate private key file
3069363408:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: ANY PRIVATE KEY
-sh-3.2# openssl s_client -cert /etc/ssl/certs/DigiCert_Trusted_Root_G4.pem
unable to load client certificate private key file
3069228240:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: ANY PRIVATE KEY
-sh-3.2# openssl s_client -key /home/root/client-key.pem
connect: Connection refused
connect:errno=111
-sh-3.2# openssl s_client -keyform /home/root/client-key.pem
connect: Connection refused
connect:errno=111
-sh-3.2# openssl s_client -pass /home/root/client-key.pem
Invalid password argument "/home/root/client-key.pem"
Error getting password

Do i have to install a certificate and if so how this can be done?

[markrad]

Hi @amiya068,

I've never researched this but I have noticed on some devices OpenSSL will ignore the directory that is apparently compiled in as the root certificate directory. I can only assume this is some other configuration option of which I am not aware.

Since you are using AMQP please try the s_client command against port 5671 rather than 443. This will ensure you don't have a firewall issue. If you want to try the -CAfile option on s_client you can download the root certificate from https://ssl-tools.net/certificates/d4de20d05e66fc53fe1a50882c78db2852cae474.pem. This certificate is the same as the first certificate found in the array in certs.c at https://github.com/Azure/azure-iot-sdk-c/blob/master/certs/certs.c.

Only pass the first certificate to the TrustedCerts option i.e. delete all of the certificates beyond the first.

Mark Radbourne MSFT

[markrad]

Hi @amiya068,

I have closed this issue as stale. If you are still experiencing problems please reopen it or open a new issue.

Thank you.

Mark Radbourne MSFT