Unable to connect MxChip DevKit as leaf device to IoT Edge #871

wickste · 2019-02-21T17:10:07Z

Configuring an MxChip devkit device as leaf device for IoT Edge I am unable to make a successful connection. I am following this guide: https://docs.microsoft.com/en-us/azure/iot-edge/how-to-connect-downstream-device#use-certificates-with-azure-iot-sdks

As a datapoint, after making the (undesirable) firmware change to disable verification in mbedtls, the connection works successfully:
mbedtls_ssl_conf_authmode(&tls_io_instance->config, MBEDTLS_SSL_VERIFY_NONE);

Expected Behavior

Device should connect as leaf deviceto IoT Edge. I have been successful with the same steps using Windows (using .NET SDK) and Ubuntu 18.04 (using C SDK as well as .NET SDK) as leaf devices.

Current Behavior

Device attempts to connect to the Edge gateway, but it appears the TLS handshake is failing.

Steps to Reproduce

Create a Azure IoT Edge on Ubuntu VM in Azure
Open all the required ports needed for a gateway scenario
Generate certificates as described in the docs
Using VSCode, create a new "Getting Started" project for MxChip using the Azure IoT Tools extension
Append "GatewayHostName=" to the connection string
Add the certification from step 3 to the project
Deploy and run on the MxChipdevice

Result: device attempts to connect to the gateway, but the TLS handshake fails

Context (Environment)

MxChip devkit with "Ubuntu Server 16.04 LTS + Azure IoT Edge runtime" VM in Azure

Device (Host) Operating System

Ubuntu 16.04 LTS

Architecture

amd64

Container Operating System

Linux

Runtime Versions

iotedged

iotedge 1.0.6.1

Edge Agent

1.0

Edge Hub

1.0

Docker

3.0.3, build 48bd4c6d

Logs

from iotedge logs edgeHub -f:

2019-02-17 19:33:09.246 +00:00 [WRN] - TLS handshake failed., System.IO.IOException: Channel is closed, 5bd3d2a6

Additional Information

The text was updated successfully, but these errors were encountered:

veyalla · 2019-02-21T18:16:26Z

This is a known issue that @ArthurMa1978 from devkit team is looking into.

ArthurMa1978 · 2019-02-27T05:57:52Z

@veyalla, from devkit side, the issue is coming from the TLS handshake step when client want to verify the cert list from server side (IoT Edge) because the cert list is not ordered which not follow TLS v1.2.
Here is the cert list, you can see the last one should be the second one:

   version: 3, type: 2340, ca 0, pathLen 0
   issure:  iotedged workload ca
   subject: 192.168.137.77

   version: 3, type: 260, ca 1, pathLen 0
   issure:  Azure IoT Hub CA Cert Test Only
   subject: Azure IoT Hub Intermediate Cert Test Only

   version: 3, type: 260, ca 1, pathLen 0
   issure:  Azure IoT Hub Intermediate Cert Test Only
   subject: 192.192.137.77.ca

   version: 3, type: 260, ca 1, pathLen 1
   issure:  192.192.137.77.ca
   subject: iotedged workload ca

The TLS 1.3 have the flexibility to support arbitrary orderings, but unfortunately mbedtls only support TLS v1.2.

myagley · 2019-02-27T16:38:30Z

This is for both MQTT and AMQP?

wickste · 2019-02-27T17:01:46Z

@myagley I have only tested with MQTT. Best I can tell, the MxChip DevKit-SDK port only supports MQTT.

ArthurMa1978 · 2019-02-27T23:22:31Z

Yes, only MQTT.

myagley · 2019-02-27T23:35:49Z

I just ran openssl s_client -connect localhost:8883 -showcerts on all of the protocols (8883, 5671, and 443) and am seeing the out of order certs in all cases. I'm following up with the dotnet core team to see if this is an issue with how we are installing these CA certs in the root store or if there is a bug in the dotnet core runtime itself.

ArthurMa1978 · 2019-02-28T00:20:15Z

@myagley
right, the openssl return the same result, that the server cert list is not a sequence chain.

myagley · 2019-02-28T00:22:12Z

After talking to the dotnet team, it appears like it is an issue in the dotnet core runtime on linux. I'll keep this issue updated when we know more about a fix.

ArthurMa1978 · 2019-02-28T03:13:45Z

Link to https://github.com/dotnet/corefx/issues/35640

ArthurMa1978 · 2019-03-01T01:06:15Z

Before dotnet core fix the order issue, not sure if IoT Edge can short the number of server cert return to client during the handshake?

ArthurMa1978 · 2019-03-15T05:54:06Z

To un-block this case quickly, I have sent a PR #2491 to mbedtls that can sort the cert chain before verify.

myagley · 2019-03-18T15:36:29Z

Thanks @ArthurMa1978

myagley · 2019-06-05T15:14:21Z

A release candidate for 1.0.8 was just released. You can try it out using the 1.0.8-rc1 tag on your images. Packages for the daemon are located here: https://github.com/Azure/azure-iotedge/releases/tag/1.0.8-rc1

stevebus · 2019-06-05T16:15:20Z

ok, I'll try it when I get a chance (probably next week)

wickste · 2019-06-05T16:28:30Z

Just a note here that this scenario already works now even with 1.07 as long as you use the latest MxChip firmware, because a mitigation has been submitted. So in order to truly verify the bug is fixed correctly in the runtime you will need to use a device/firmware that still exposes the bug (e.g. an older MxChip firmware)

stevebus · 2019-06-05T16:30:39Z

I have one :-) thx for the heads up...

myagley · 2019-06-05T22:52:07Z

Thanks for the heads up @wickste!

stevebus · 2019-06-25T19:18:48Z

FYI - Sorry for the delays. I won't bore you with excuses.. I just tried this with my MXChip and Edge 1.0.7.1 and @ArthurMa1978 's excellent example at https://github.com/IoTDevEnvExamples/DevKit2IoTEdge and it worked fine. MXChip was able to connect over MQTTS to my IoT Edge box and start sending data..

interestingly, I notice that the "depth" part of the iot edge cert dump (from openssl connect command is in order), but the 'certificate' chain isn't.. I assume it's the first part that counts from a TLS perspective?

stevebus@sdbubuntu2:~$ openssl s_client -connect sdbubuntu2.centralus.cloudapp.azure.com:8883
CONNECTED(00000005)
depth=4 CN = Azure_IoT_Hub_CA_Cert_Test_Only
verify return:1
depth=3 CN = Azure_IoT_Hub_Intermediate_Cert_Test_Only
verify return:1
depth=2 CN = mygateway.ca
verify return:1
depth=1 CN = iotedged workload ca
verify return:1
depth=0 CN = sdbubuntu2.centralus.cloudapp.azure.com
verify return:1

Certificate chain
0 s:CN = sdbubuntu2.centralus.cloudapp.azure.com
i:CN = iotedged workload ca
1 s:CN = Azure_IoT_Hub_Intermediate_Cert_Test_Only
i:CN = Azure_IoT_Hub_CA_Cert_Test_Only
2 s:CN = mygateway.ca
i:CN = Azure_IoT_Hub_Intermediate_Cert_Test_Only
3 s:CN = iotedged workload ca
i:CN = mygateway.ca

Either way, looks like this is fixed now, assuming the getting started sample in MXChip actually does validate the edgeHub TLS cert. Thanks for all the work on it, guys!

ericwolz · 2019-07-25T23:54:39Z

I updated my hub to 1.0.8 and tried to validate it against the MXChip board version 1.5.1 (does not contain the mbedtls workaround).

This issue still repros for me.

myagley · 2019-07-26T15:06:58Z

@ArthurMa1978 how did we figure out that this was due to certificate ordering issues to begin with? Is it possible to run the same command/test with our 1.0.8 version to see if it has changed?

ericwolz · 2019-08-01T20:30:48Z

@myagley look above, he used
openssl s_client -connect sdbubuntu2.centralus.cloudapp.azure.com:8883

ericwolz · 2019-08-01T22:07:04Z

This is my cert chain. Does this look correct?

CONNECTED(00000003)
depth=3 CN = Azure IoT CA TestOnly Intermediate CA
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
 0 s:/CN=192.168.0.128
   i:/CN=iotedged workload ca
 1 s:/CN=iotedged workload ca
   i:/CN=192.168.0.128.ca
 2 s:/CN=192.168.0.128.ca
   i:/CN=Azure IoT CA TestOnly Intermediate CA
 3 s:/CN=Azure IoT CA TestOnly Intermediate CA
   i:/CN=Azure IoT CA TestOnly Root CA

stevebus · 2019-08-02T15:52:35Z

yes, that chain looks correct... as I mentioned earlier above, the odd thing was that one of the lists (the top one) was in the correct order, but the bottom one in the openssl command, which is labeled certificate chain, was not).. I just re-tested with a fresh 1.0.8 box, and both lists are now in the correct order (see output below), so I assume that something changed between 1.0.7.1 and 1.0.8. The only MXChip test I've done lately was based on ArturMa's sample referenced above (which I assume has the MBED fix in it), so I can't say if it works without the fix. (primarily because I can't get the iot-central-firmware MXChip sample to build any more)

CONNECTED(00000003)
depth=4 CN = Azure_IoT_Hub_CA_Cert_Test_Only
verify return:1
depth=3 CN = Azure_IoT_Hub_Intermediate_Cert_Test_Only
verify return:1
depth=2 CN = device_ca_1.ca
verify return:1
depth=1 CN = iotedged workload ca
verify return:1
depth=0 CN = sdb_edge_01
verify return:1

Certificate chain
0 s:/CN=sdb_edge_01
i:/CN=iotedged workload ca
1 s:/CN=iotedged workload ca
i:/CN=device_ca_1.ca
2 s:/CN=device_ca_1.ca
i:/CN=Azure_IoT_Hub_Intermediate_Cert_Test_Only
3 s:/CN=Azure_IoT_Hub_Intermediate_Cert_Test_Only
i:/CN=Azure_IoT_Hub_CA_Cert_Test_Only

myagley · 2019-08-02T17:33:26Z

Yes, we picked up a newer version of the dotnet runtime which has the fix for the cert ordering issue in 1.0.8.

Note, that using an IP address for the hostname is unlikely to work properly. This will not validate properly during TLS. You will need to use a real hostname for the device and make sure that the mxchip can resolve the hostname to ip address properly.

stevebus · 2019-08-02T18:04:57Z

actually, using IP address works fine, as long as you also use the IP address in the hostname parameter in config.yaml. Edge will generate the edge TLS cert with the IP address as the CN and the cert verification works fine.. at least for every client I've tried so far with it (including the MXChip, which is the first place I tried it because i didn't have access to a DNS server :-) )

ericwolz · 2019-08-02T18:11:51Z

FYI: IP address cert is working fine for me.

Trying to repro the issue on 1.0.7 and 1.0.6, but it looks like the edge container picks up the current version of the dotnet runtime so the cert order are correct for these version now.

With version 1.6.2 and higher of the MXChip, everything is working correctly.

myagley · 2019-08-02T21:41:32Z

Interesting about the IP address. I didn't realize that would work.

IoT Edge bundles the runtime inside the container so it;s not possible to pick it up from anywhere. The only version with the fixed version of the dotnet runtime is 1.0.8.

github-actions · 2019-11-04T00:03:25Z

This issue is being marked as stale because it has been open for 30 days with no activity.

lt72 · 2019-11-19T17:54:59Z

Closing this issue as was fixed in 1.0.8. Please re-open as needed.

myagley mentioned this issue Feb 28, 2019

Unable to connect Azure Sphere DevKit as leaf device to IoT Edge #872

Closed

myagley added the bug Something isn't working label Mar 1, 2019

ArthurMa1978 mentioned this issue Mar 28, 2019

Support un-sorted server cert chain during TLS handshake microsoft/devkit-sdk#979

Closed

myagley added 1.0.8 Targeted for 1.0.8 release checkedin labels May 29, 2019

github-actions bot added the no-issue-activity label Nov 4, 2019

lt72 closed this as completed Nov 19, 2019

yophilav added customer-reported iotedge labels Mar 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to connect MxChip DevKit as leaf device to IoT Edge #871

Unable to connect MxChip DevKit as leaf device to IoT Edge #871

wickste commented Feb 21, 2019 •

edited

Loading

veyalla commented Feb 21, 2019

ArthurMa1978 commented Feb 27, 2019 •

edited

Loading

myagley commented Feb 27, 2019

wickste commented Feb 27, 2019 •

edited

Loading

ArthurMa1978 commented Feb 27, 2019 via email •

edited

Loading

myagley commented Feb 27, 2019

ArthurMa1978 commented Feb 28, 2019

myagley commented Feb 28, 2019

ArthurMa1978 commented Feb 28, 2019 •

edited

Loading

ArthurMa1978 commented Mar 1, 2019

ArthurMa1978 commented Mar 15, 2019

myagley commented Mar 18, 2019

myagley commented Jun 5, 2019

stevebus commented Jun 5, 2019

wickste commented Jun 5, 2019

stevebus commented Jun 5, 2019

myagley commented Jun 5, 2019

stevebus commented Jun 25, 2019

ericwolz commented Jul 25, 2019

myagley commented Jul 26, 2019

ericwolz commented Aug 1, 2019

ericwolz commented Aug 1, 2019

stevebus commented Aug 2, 2019

myagley commented Aug 2, 2019

stevebus commented Aug 2, 2019

ericwolz commented Aug 2, 2019

myagley commented Aug 2, 2019

github-actions bot commented Nov 4, 2019

lt72 commented Nov 19, 2019

Unable to connect MxChip DevKit as leaf device to IoT Edge #871

Unable to connect MxChip DevKit as leaf device to IoT Edge #871

Comments

wickste commented Feb 21, 2019 • edited Loading

Expected Behavior

Current Behavior

Steps to Reproduce

Context (Environment)

Device (Host) Operating System

Architecture

Container Operating System

Runtime Versions

iotedged

Edge Agent

Edge Hub

Docker

Logs

Additional Information

veyalla commented Feb 21, 2019

ArthurMa1978 commented Feb 27, 2019 • edited Loading

myagley commented Feb 27, 2019

wickste commented Feb 27, 2019 • edited Loading

ArthurMa1978 commented Feb 27, 2019 via email • edited Loading

myagley commented Feb 27, 2019

ArthurMa1978 commented Feb 28, 2019

myagley commented Feb 28, 2019

ArthurMa1978 commented Feb 28, 2019 • edited Loading

ArthurMa1978 commented Mar 1, 2019

ArthurMa1978 commented Mar 15, 2019

myagley commented Mar 18, 2019

myagley commented Jun 5, 2019

stevebus commented Jun 5, 2019

wickste commented Jun 5, 2019

stevebus commented Jun 5, 2019

myagley commented Jun 5, 2019

stevebus commented Jun 25, 2019

ericwolz commented Jul 25, 2019

myagley commented Jul 26, 2019

ericwolz commented Aug 1, 2019

ericwolz commented Aug 1, 2019

stevebus commented Aug 2, 2019

CONNECTED(00000003) depth=4 CN = Azure_IoT_Hub_CA_Cert_Test_Only verify return:1 depth=3 CN = Azure_IoT_Hub_Intermediate_Cert_Test_Only verify return:1 depth=2 CN = device_ca_1.ca verify return:1 depth=1 CN = iotedged workload ca verify return:1 depth=0 CN = sdb_edge_01 verify return:1

myagley commented Aug 2, 2019

stevebus commented Aug 2, 2019

ericwolz commented Aug 2, 2019

myagley commented Aug 2, 2019

github-actions bot commented Nov 4, 2019

lt72 commented Nov 19, 2019

wickste commented Feb 21, 2019 •

edited

Loading

ArthurMa1978 commented Feb 27, 2019 •

edited

Loading

wickste commented Feb 27, 2019 •

edited

Loading

ArthurMa1978 commented Feb 27, 2019 via email •

edited

Loading

ArthurMa1978 commented Feb 28, 2019 •

edited

Loading

CONNECTED(00000003)
depth=4 CN = Azure_IoT_Hub_CA_Cert_Test_Only
verify return:1
depth=3 CN = Azure_IoT_Hub_Intermediate_Cert_Test_Only
verify return:1
depth=2 CN = device_ca_1.ca
verify return:1
depth=1 CN = iotedged workload ca
verify return:1
depth=0 CN = sdb_edge_01
verify return:1