Unauthorized error when provision devices in Enrollment Group

[michael-chi]

• OS, version, SKU and CPU architecture used: Ubuntu 18.04
• Application's .NET Target Framework : netcoreapp2.2
• Device: Desktop
• SDK version used:
  Microsoft.Azure.Devices.Client - 1.20.3
  Microsoft.Azure.Devices.Provisioning.Client - 1.4.0
  Microsoft.Azure.Devices.Provisioning.Transport.Amqp - 1.1.9

Description of the issue:

I am trying to setup Enrollment Group in my environment with self-signed certificate and is having issue connecting device to my IoT Hub through DPS. Kindly point me a direction.

Code sample exhibiting the issue:

Repro Steps

• Create Certificates

Follow this instruction to create required certificates. Below are all my steps

chmod 700 certGen.sh

# Create Root and Intermediate CA
./certGen.sh create_root_and_intermediate

# Upload Root CA to IoT DPS
az iot dps certificate create --dps-name michi-dps-20190723 --resource-group michi-dps-rg --name dps20190725root --path ./certs/azure-iot-test-only.root.ca.cert.pem

# Get ETAG
az iot dps certificate list --dps-name michi-dps-20190723 -g michi-dps-rg

# Generate Validation Code for Root CA
az iot dps certificate generate-verification-code -g michi-dps-rg --dps-name michi-dps-20190723 -n dps20190725root -e AAAAAAD6w0Y=

# Generate validation CA for Root CA
./certGen.sh create_verification_certificate <VALIDATION_CODE>

# Get latest ETAG again
az iot dps certificate list --dps-name michi-dps-20190723 -g michi-dps-rg

# Verify Root CA
az iot dps certificate verify --dps-name michi-dps-20190723 -g michi-dps-rg --name dps20190725root --path ./certs/verification-code.cert.pem -e AAAAAAD7qnQ==

# Verify was successful, this can be confirmed via portal as well.

• Once verified, generate device certificate

# Create Device CA
rm ./certs/new-device.cert.pem
./certGen.sh create_device_certificate deviceid001

• Create Enrollment Group via Portal

I use portal to create an Enrollment Group and choose the root certificate I uploaded. with "CA Certificate" as Certificate Type.

• I download X509Sample code from here

• Copy generated new-device.cert.pfx to project folder then

• Modify Program.cs to read my pfx file

private static string s_certificateFileName = "new-device.cert.pfx";

• According to project readme.md, I change my codes as below

#if false
    X509Certificate2 certificate = LoadProvisioningCertificate();
    using (var security = new SecurityProviderX509Certificate(certificate))
#else
    var myCertificate = new X509Certificate2(s_certificateFileName, "1234");
    var myChain = new X509Certificate2Collection();
    
    // Comment out the below line if you do not have a .p7b file (e.g. if you generated certificates using the tool below)
    //myChain.Import("myChain.p7b");
    
    using (var security = new SecurityProviderX509Certificate(myCertificate, myChain))
#endif

• Compile and Run

dotnet restore
dotnet build
dotnet run

Console log of the issue:

RegistrationID = deviceid001
ProvisioningClient RegisterAsync . . .
Unhandled Exception: Microsoft.Azure.Devices.Provisioning.Client.ProvisioningTransportException: {"errorCode":401002,"trackingId":"4fdd0060-3b0a-42e7-9066-5badf3bef16c","message":"Unauthorized","timestampUtc":"2019-07-25T09:15:03.0636061Z"}
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerAmqp.ValidateOutcome(Outcome outcome)
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerAmqp.RegisterDeviceAsync(AmqpClientConnection client, String correlationId, DeviceRegistration deviceRegistration)
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerAmqp.RegisterAsync(ProvisioningTransportRegisterMessage message, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Provisioning.Client.Samples.ProvisioningDeviceClientSample.RunSampleAsync() in /home/azureuser/dps/x509sample/Common/ProvisioningDeviceClientSample.cs:line 31
   at Microsoft.Azure.Devices.Provisioning.Client.Samples.Program.Main(String[] args) in /home/azureuser/dps/x509sample/Program.cs:line 82

[michael-chi]

Hi Team,
Here's how I get things work. I think there should be something wrong in the generated certificates.

Assume IoT DPS and IoT Hub are already created

• Resource Group: michi-dps-201907260-rg
• DPS: michi-dps-20190726
• IoT Hub: michi-dps-eastasia-001
  DPS and IoT Hub already linked

Spec

• OS: Ubuntu 18.04
• OpenSSL: 1.1.1 11 Sep 2018
• azure-cli: 2.0.69
  • azure-cli-iot-ext: 0.7.1

# Clone and copy everything here to working folder
# https://github.com/Azure/azure-iot-sdk-c/tree/master/tools/CACertificates

# Create Root Certificate
# Set $ENV:OPENSSL_CONF="C:\Program Files\openssl-1.1.1c-win64-mingw\openssl.cnf"
# export $ENV:OPENSSL_CONF="/mnt/c/Program Files/openssl-1.1.1c-win64-mingw/openssl.cnf"

cd tools
chmod 700 certGen.sh
chmod 700 certGenEx.sh

# Create Root and Intermediate CA
./certGen.sh create_root_and_intermediate

## Chain
# Concat root and intermediate certificate
cat certs/azure-iot-test-only.intermediate.cert.pem certs/azure-iot-test-only.root.ca.cert.pem > certs/ca-full-chain.cert.pem

# Upload ca-full-chain.cert.pem to Azure DPS
# az iot dps certificate create --dps-name michi-dps-20190726 --resource-group # michi-dps-20190726-rg --name dps20190726chain --path ./certs/ca-full-chain.cert.pem

# Manually upload ca-ful-chain.cert.pem to DPS and name it dps20190726chain

# Get Latest ETAG (AAAAAAD8czU=)
az iot dps certificate show -n dps20190726chain --dps-name michi-dps-20190726 -g michi-dps-20190726-rg

# Generate Validation Code for Chain CA (2EF194C10D480915EBB4DAD7B3A18CCDEECF793D1FFEAFB6)
az iot dps certificate generate-verification-code -g michi-dps-20190726-rg --dps-name michi-dps-20190726 -n dps20190726chain -e <ETAG>

# Create verification certificate
./certGenEx.sh  create_intermediate_verification_certificate <VERIFICATION_CODE>

# Get Latest ETAG (AAAAAAD8c2k=)
az iot dps certificate show -n dps20190726chain --dps-name michi-dps-20190726 -g michi-dps-20190726-rg

# Verify Chain CA
az iot dps certificate verify --dps-name michi-dps-20190726 -g michi-dps-rg --name dps20190726chain --path ./certs/verification-code.cert.pem -e <ETAG>

## Create Enrollment Group with GeoLatency allocation policy
az iot dps enrollment-group create --dps-name michi-dps-20190726 -g michi-dps-20190726-rg --enrollment-id dpseastasia --root-ca-name dps20190726chain  --ap geolatency

# Create Device CA
rm ./certs/new-device.cert.pem
./certGen.sh create_device_certificate deviceid001

certGenEx.sh

## Copyright (c) Microsoft. All rights reserved.
## Licensed under the MIT license. See LICENSE file in the project root for full license information.

###############################################################################
# This script demonstrates creating X.509 certificates for an Azure IoT Hub
# CA Cert deployment.
#
# These certs MUST NOT be used in production.  It is expected that production
# certificates will be created using a company's proper secure signing process.
# These certs are intended only to help demonstrate and prototype CA certs.
###############################################################################

root_ca_dir="."
home_dir="."
algorithm="genrsa"
COUNTRY="US"
STATE="WA"
LOCALITY="Redmond"
ORGANIZATION_NAME="My Organization"
root_ca_password="1234"
key_bits_length="4096"
days_till_expire=30
ca_chain_prefix="azure-iot-test-only.chain.ca"
intermediate_ca_dir="."
openssl_root_config_file="./openssl_root_ca.cnf"
openssl_intermediate_config_file="./openssl_device_intermediate_ca.cnf"
intermediate_ca_password="1234"
root_ca_prefix="azure-iot-test-only.root.ca"
intermediate_ca_prefix="azure-iot-test-only.intermediate"

function makeCNsubject()
{
    local result="/CN=${1}"
    case $OSTYPE in
        msys|win32) result="/${result}"
    esac
    echo "$result"
}

function warn_certs_not_for_production()
{
    tput smso
    tput setaf 3
    echo "Certs generated by this script are not for production (e.g. they have hard-coded passwords of '1234'."
    echo "This script is only to help you understand Azure IoT Hub CA Certificates."
    echo "Use your official, secure mechanisms for this cert generation."
    echo "Also note that these certs will expire in ${days_till_expire} days."
    tput sgr0
}
###############################################################################
# Generate a Certificate for a device using specific openssl extension and
# signed with either the root or intermediate cert.
###############################################################################
function generate_device_certificate_common()
{
    local common_name="${1}"
    local device_prefix="${2}"
    local certificate_dir="${3}"
    local ca_password="${4}"
    local server_pfx_password="1234"
    local password_cmd=" -passin pass:${ca_password} "
    local openssl_config_file="${5}"
    local openssl_config_extension="${6}"
    local cert_type_diagnostic="${7}"

    echo "Creating ${cert_type_diagnostic} Certificate"
    echo "----------------------------------------"
    cd ${home_dir}

    openssl ${algorithm} \
            -out ${certificate_dir}/private/${device_prefix}.key.pem \
            ${key_bits_length}
    [ $? -eq 0 ] || exit $?
    chmod 444 ${certificate_dir}/private/${device_prefix}.key.pem
    [ $? -eq 0 ] || exit $?

    echo "Create the ${cert_type_diagnostic} Certificate Request"
    echo "----------------------------------------"
    openssl req -config ${openssl_config_file} \
        -key ${certificate_dir}/private/${device_prefix}.key.pem \
        -subj "$(makeCNsubject "${common_name}")" \
        -new -sha256 -out ${certificate_dir}/csr/${device_prefix}.csr.pem
    [ $? -eq 0 ] || exit $?

    openssl ca -batch -config ${openssl_config_file} \
            ${password_cmd} \
            -extensions "${openssl_config_extension}" \
            -days ${days_till_expire} -notext -md sha256 \
            -in ${certificate_dir}/csr/${device_prefix}.csr.pem \
            -out ${certificate_dir}/certs/${device_prefix}.cert.pem
    [ $? -eq 0 ] || exit $?
    chmod 444 ${certificate_dir}/certs/${device_prefix}.cert.pem
    [ $? -eq 0 ] || exit $?

    echo "Verify signature of the ${cert_type_diagnostic}" \
         " certificate with the signer"
    echo "-----------------------------------"
    openssl verify \
            -CAfile ${certificate_dir}/certs/${ca_chain_prefix}.cert.pem \
            ${certificate_dir}/certs/${device_prefix}.cert.pem
    [ $? -eq 0 ] || exit $?

    echo "${cert_type_diagnostic} Certificate Generated At:"
    echo "----------------------------------------"
    echo "    ${certificate_dir}/certs/${device_prefix}.cert.pem"
    echo ""
    openssl x509 -noout -text \
            -in ${certificate_dir}/certs/${device_prefix}.cert.pem
    [ $? -eq 0 ] || exit $?
    echo "Create the ${cert_type_diagnostic} PFX Certificate"
    echo "----------------------------------------"
    openssl pkcs12 -in ${certificate_dir}/certs/${device_prefix}.cert.pem \
            -inkey ${certificate_dir}/private/${device_prefix}.key.pem \
            -password pass:${server_pfx_password} \
            -export -out ${certificate_dir}/certs/${device_prefix}.cert.pfx
    [ $? -eq 0 ] || exit $?
    echo "${cert_type_diagnostic} PFX Certificate Generated At:"
    echo "--------------------------------------------"
    echo "    ${certificate_dir}/certs/${device_prefix}.cert.pfx"
    [ $? -eq 0 ] || exit $?
}

###############################################################################
# Generate a certificate for a leaf device
# signed with either the root or intermediate cert.
###############################################################################
function generate_leaf_certificate()
{
    local common_name="${1}"
    local device_prefix="${2}"
    local certificate_dir="${3}"
    local ca_password="${4}"
    local openssl_config_file="${5}"

    generate_device_certificate_common "${common_name}" "${device_prefix}" \
                                       "${certificate_dir}" "${ca_password}" \
                                       "${openssl_config_file}" "server_cert" \
                                       "Leaf Device"
}

###############################################################################
#  Creates required directories and removes left over cert files.
#  Run prior to creating Root CA; after that these files need to persist.
###############################################################################
function prepare_filesystem()
{
    if [ ! -f ${openssl_root_config_file} ]; then
        echo "Missing file ${openssl_root_config_file}"
        exit 1
    fi

    if [ ! -f ${openssl_intermediate_config_file} ]; then
        echo "Missing file ${openssl_intermediate_config_file}"
        exit 1
    fi

    rm -rf csr
    rm -rf private
    rm -rf certs
    rm -rf intermediateCerts
    rm -rf newcerts

    mkdir -p csr
    mkdir -p private
    mkdir -p certs
    mkdir -p intermediateCerts
    mkdir -p newcerts

    rm -f ./index.txt
    touch ./index.txt

    rm -f ./serial
    echo 01 > ./serial
}

###############################################################################
# Generates a root and intermediate certificate for CA certs.
###############################################################################
function initial_cert_generation()
{
    prepare_filesystem
    generate_root_ca
    generate_intermediate_ca
}


###############################################################################
# Generates a certificate for verification, chained directly to the root.
###############################################################################
function generate_intermediate_verification_certificate()
{
    if [$# -ne 1]; then
        echo "Usage: <subjectName>"
        exit 1
    fi

    rm -f ./private/verification-code.key.pem
    rm -f ./certs/verification-code.cert.pem
    generate_leaf_certificate "${1}" "verification-code" \
                              ${intermediate_ca_dir} ${intermediate_ca_password} \
                              ${openssl_intermediate_config_file}
}

if [ "${1}" == "create_intermediate_verification_certificate" ]; then
    generate_intermediate_verification_certificate "${2}"
else
    echo "Usage: create_intermediate_verification_certificate"
    exit 1
fi

warn_certs_not_for_production

[prmathur-microsoft]

You could chose one of the following two options :

1. Upload intermediate only and do Proof of possession on that
   or

2. Include intermediate in tls handshake
   Any of the above two options can be valid options.

[prmathur-microsoft]

Closing this as per comment above. Feel free to reopen this if you have further questions.

[az-iot-builder-01]

@michael-chi, @prmathur-microsoft, thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey

[michael-chi]

Hi Team,
Could you elaborate #2 ?
Option 1 doesn't work for me in same environment. below are my step

Method 1

1. Upload intermediate cert to DPS via portal and generate verification code
2. run below command to generate verification cert

./certGen.sh create_verification_certificate <VERIFICATION_CODE>

3. Upload verification cert and verify but this will fail

Method 2

1. Upload root cert and successfully verify it.
2. Upload intermediate and repeat above steps to verify it. This is failed.

Method 3

1. Create a new enrollment group, this time choose "Intermediate Cert" and upload intermediate cert.
2. Generate device cert and try to provision and fail.

Error

Found certificate: DB4860FE4ED613ACBAB497914F97056CC2C15A53 CN=michi20190804001; PrivateKey: True
Using certificate DB4860FE4ED613ACBAB497914F97056CC2C15A53 CN=michi20190804001
RegistrationID = michi20190804001
ProvisioningClient RegisterAsync . . .
Unhandled Exception: Microsoft.Azure.Devices.Provisioning.Client.ProvisioningTransportException: {"errorCode":401002,"trackingId":"0438d617-4d00-4cec-b2f8-48e14e2a24b0","message":"Unauthorized","timestampUtc":"2019-08-04T07:55:35.6038985Z"}
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerAmqp.ValidateOutcome(Outcome outcome)
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerAmqp.RegisterDeviceAsync(AmqpClientConnection client, String correlationId, DeviceRegistration deviceRegistration)
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerAmqp.RegisterAsync(ProvisioningTransportRegisterMessage message, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Provisioning.Client.Samples.ProvisioningDeviceClientSample.RunSampleAsync() in /home/azureuser/dps/x509sample/Common/ProvisioningDeviceClientSample.cs:line 31
   at Microsoft.Azure.Devices.Provisioning.Client.Samples.Program.Main(String[] args) in /home/azureuser/dps/x509sample/Program.cs:line 82

[prmathur-microsoft]

For #2 You need to add intermediate to the the trusted store on your device.
For method 3 - Did you verify with PoP after uploading intermediate?

[michael-chi]

For #2 You need to add intermediate to the the trusted store on your device.
For method 3 - Did you verify with PoP after uploading intermediate?

yes, I verify PoP after uploading it...the verify fails always...

[prmathur-microsoft]

Does your PoP verification of the intermediate fails ? or provisioning fails?

[prmathur-microsoft]

I would like to close this as I don't have any update from you in long time.
Feel free to reopen when you get more data.