Automatic EST certificate retrieval does not work on Ubuntu 22.04

[gcobanhelin]

Expected Behavior

Using a valid bootstrap certificate and the right config file, IoT Edge should automatically get the device-id and est-id certificate from our EST endpoint and then provision with DPS. We have a working setup on Ubuntu 18.04, Ubuntu 20.04 and Debian Buster/Bullseye.

Current Behavior

IoT Edge cannot get any certificate, /var/lib/aziot/certd/certs is empty. iotedge system logs gives an error (see logs) and edge agent is not running.

Steps to Reproduce

1. Get a valid bootstrap certificate from your EST endpoint which includes the CA cert
2. Use the following config file with valid values for dps_id_scope, est_endpoint and BOOTSTRAP_FOLDER:

[provisioning]
source = "dps"
global_endpoint = "https://global.azure-devices-provisioning.net"
id_scope = "{{dps_id_scope}}"

[provisioning.attestation]
method = "x509"
registration_id = "device_id"

[provisioning.attestation.identity_cert]
method = "est"
common_name = "device_id"
[cert_issuance.est.urls]
default = "{{est_endpoint}}"

[cert_issuance.est.auth]
bootstrap_identity_cert = "file://BOOTSTRAP_FOLDER/bootstrap_chain.pem"
bootstrap_identity_pk = "file://BOOTSTRAP_FOLDER/bootstrap.key"

[provisioning.attestation.identity_cert.auto_renew]
rotate_key = true
threshold = "80%"
retry = "4%"

[cert_issuance.est.identity_auto_renew]
rotate_key = true
threshold = "80%"
retry = "4%"

4. sudo iotedge config apply
5. sudo iotedge system logs
6. Now the error comes up

Context (Environment)

Output of iotedge check

Click here

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ production readiness: identity certificates expiry - OK
× production readiness: EST identity and bootstrap certificates expiry - Error
    x509 identity certificate is neither preloaded nor configured to be dynamically issued, and thus cannot be used.
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
× read all preloaded certificates from the Certificates Service - Error
    could not load cert with ID "aziot-edged-trust-bundle"

    Caused by:
        parameter "id" has an invalid value
        caused by: not found
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
√ host can connect to and perform TLS handshake with DPS endpoint - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
× configuration has correct URIs for daemon mgmt endpoint - Error
    SocketError - SocketErrorCode (TimedOut) : Operation timed out
    One or more errors occurred. (Got bad response: )
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
× production readiness: Edge Agent's storage directory is persisted on the host filesystem - Error
    Could not check current state of edgeAgent container
× production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error
    Could not check current state of edgeHub container
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
23 check(s) succeeded.
5 check(s) raised warnings. Re-run with --verbose for more details.
5 check(s) raised errors. Re-run with --verbose for more details.
7 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

Device Information

• Host OS [e.g. Ubuntu 18.04, Windows Server IoT 2019]: Ubuntu 22.04
• Architecture [e.g. amd64, arm32, arm64]: amd64
• Container OS [e.g. Linux containers, Windows containers]: Linux

Runtime Versions

• aziot-edged [run iotedge version]: 1.4.9
• aziot-identity-service: 1.4.3
• Edge Agent [image tag (e.g. 1.0.0)]: not running
• Edge Hub [image tag (e.g. 1.0.0)]: not running
• Docker/Moby [run docker version]: 20.10.23+azure-2

Logs

aziot-edged logs

Mar 29 12:24:32 vm-ansible-ubuntu22-x64 aziot-certd[6999]: 2023-03-29T12:24:32Z [ERR!] - !!! internal error
Mar 29 12:24:32 vm-ansible-ubuntu22-x64 aziot-certd[6999]: 2023-03-29T12:24:32Z [ERR!] - !!! caused by: could not create cert
Mar 29 12:24:32 vm-ansible-ubuntu22-x64 aziot-certd[6999]: 2023-03-29T12:24:32Z [ERR!] - !!! caused by: cert "device-id" is configured to be issued by EST, but neither EST identity nor EST bootstrap identity could be obtained: error trying to connect: error:0A080006:SSL routines:ssl_generate_param_group:EVP lib:../ssl/s3_lib.c:4735:, error:0A00013A:SSL routines:tls_process_ske_ecdhe:unable to find ecdh parameters:../ssl/statem/statem_clnt.c:2140:
Mar 29 12:24:32 vm-ansible-ubuntu22-x64 aziot-certd[6999]: 2023-03-29T12:24:32Z [INFO] - --> 500 {"content-type": "application/json"}
Mar 29 12:24:32 vm-ansible-ubuntu22-x64 aziot-identityd[7024]: 2023-03-29T12:24:32Z [ERR!] - service encountered an error
Mar 29 12:24:32 vm-ansible-ubuntu22-x64 aziot-identityd[7024]: 2023-03-29T12:24:32Z [ERR!] - caused by: internal error
Mar 29 12:24:32 vm-ansible-ubuntu22-x64 aziot-identityd[7024]: 2023-03-29T12:24:32Z [ERR!] - caused by: could not create certificate
Mar 29 12:24:32 vm-ansible-ubuntu22-x64 aziot-identityd[7024]: 2023-03-29T12:24:32Z [ERR!] - caused by: internal error
Mar 29 12:24:32 vm-ansible-ubuntu22-x64 aziot-identityd[7024]: 2023-03-29T12:24:32Z [ERR!] -    0: <unknown>
Mar 29 12:24:32 vm-ansible-ubuntu22-x64 aziot-identityd[7024]:    1: <unknown>
Mar 29 12:24:32 vm-ansible-ubuntu22-x64 systemd[1]: aziot-identityd.service: Main process exited, code=exited, status=1/FAILURE
Mar 29 12:24:32 vm-ansible-ubuntu22-x64 systemd[1]: aziot-identityd.service: Failed with result 'exit-code'.

Additional Information

We are often getting the timeout in 'iotedge check' for the check 'configuration has correct URIs for daemon mgmt endpoint', also for our devices that the EST issuance works on

[ggjjj]

This appears to be a known issue with openssl 3 (which is what's different about Ubuntu 22.04). We have seen this before. There's nothing we can do except wait for it to be fixed in openssl.

[jlian]

@ggjjj is this the part where you can tell it's an openssl issue?

error trying to connect: error:0A080006:SSL routines:ssl_generate_param_group:EVP lib:../ssl/s3_lib.c:4735:, error:0A00013A:SSL routines:tls_process_ske_ecdhe:unable to find ecdh parameters:../ssl/statem/statem_clnt.c:2140:

Does this mean that EST is a non-starter for Ubuntu 22.04, always?

CC @gordonwang0

[ggjjj]

@ggjjj is this the part where you can tell it's an openssl issue?

error trying to connect: error:0A080006:SSL routines:ssl_generate_param_group:EVP lib:../ssl/s3_lib.c:4735:, error:0A00013A:SSL routines:tls_process_ske_ecdhe:unable to find ecdh parameters:../ssl/statem/statem_clnt.c:2140:

Does this mean that EST is a non-starter for Ubuntu 22.04, always?

CC @gordonwang0

Gordon mentioned it as openssl issue

[ggjjj]

Once this issue is fixed openssl/openssl#20161 it should work

[bishal41]

@gcobanhelin - it looks like we have a dependency on another issue, do you want to keep this issue open or close this issue, track the dependent issue and circle back on this one as needed?

[konichi3]

@ggjjj the openSSL issue was fixed yesterday (openssl/openssl#20161). What is the next step?

[ggjjj]

With the fix, this issue should resolve as per @gordonwang0

[ggjjj]

Update: We need to wait for the next openssl release and also the updated openssl to be published to the package repositories.

[konichi3]

@gcobanhelin Closing the issue as the OpenSSL issue was fixed. The team that made the fix needs to produce the package and there is no further actions on IoT Edge team side.

[omedirk]

Writing this a a coworker of gcobanhelin, the inital starter of the topic.

Please reopen this issue as it was fixed in openSSL but not in the ubuntu release.
ubuntu 22.04 is a tier1 solution, but does in this case not work out of the box.
what is the proposed solution then?
=> do we need to do a compile of a newer openSSL?

if so, this should be on the iotedge website as an install step in my opinion.

any help appreciated.