Failed to get a certificate from EST server during x509 Provisioning when CSR subject contains additional fields (not only CN) #455

bemol38 · 2022-09-05T11:00:26Z

Hi everybody,
What I want to achieve is to provision my device to my DPS, using X509 certificate attestation, using as identity_cert a certificate issued by a Digicert EST end-point, which requires a CSR containing both a common_name (CN) and an organizational_unit (OU) fields.
To authenticate to the EST server, a pre-existing "birth" certificate (according to our own terminology) is used, the key pair of which is stored in TPM.

Therefore my config.toml template looks like below, and the env variables are substituted with the envsubst command, before aziotctl config is applied.

## DPS provisioning with X.509 certificate
[provisioning]
source = "dps"
global_endpoint = "https://global.azure-devices-provisioning.net"
id_scope = "$scope_id"
 
[provisioning.attestation]
method = "x509"
registration_id = "$registration_id"

[provisioning.attestation.identity_cert]
method = "est"
url = "$cert_issuance_est_urls"
subject = { CN = "$common_name", OU = "$organization_unit" }

[aziot_keys]
pkcs11_lib_path = "$pkcs11_lib_path"
pkcs11_base_slot = "$pkcs11_base_slot"

[cert_issuance.est]
trusted_certs = [
    "$trusted_cert",
]

[cert_issuance.est.auth]
identity_cert = "$birth_cert_path"
identity_pk = "$pkcs11_birth_key_slot"

[cert_issuance.est.urls]
default = "$cert_issuance_est_urls"

By doing so, we get the following error (see end of the log):

sept. 05 11:01:21 myself aziot-identityd[46367]: 2022-09-05T09:01:21Z [INFO] - Starting service...
sept. 05 11:01:21 myself aziot-identityd[46367]: 2022-09-05T09:01:21Z [INFO] - Version - 1.3.0
sept. 05 11:01:21 myself aziot-identityd[46367]: 2022-09-05T09:01:21Z [INFO] - Provisioning starting. Reason: Startup
sept. 05 11:01:21 myself systemd[1]: Started Azure IoT Keys Service.
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - Starting service...
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - Version - 1.3.0
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - Loaded libaziot-keys with version 0x02010000
sept. 05 11:01:21 myself aziot-keyd[46384]: WARNING:fapi:src/tss2-fapi/api/Fapi_List.c:226:Fapi_List_Finish() Profile of path not provisioned: /HS/SRK
sept. 05 11:01:21 myself aziot-keyd[46384]: ERROR:fapi:src/tss2-fapi/api/Fapi_List.c:81:Fapi_List() ErrorCode (0x00060034) Entities_List
sept. 05 11:01:21 myself aziot-keyd[46384]: ERROR: Listing FAPI token objects failed.
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - Starting server...
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - <-- GET /keypair/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [ERR!] - invalid parameter "id": not found
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - !!! a parameter has an invalid value
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - --> 400 {"content-type": "application/json"}
sept. 05 11:01:21 myself systemd[1]: Started Azure IoT Certificates Service.
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - Starting service...
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - Version - 1.3.0
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - Certificate est-id will be auto-renewed. Next renewal at 2023-05-31T06:41:52+00:00.
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - Starting server...
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - <-- GET /certificates/device-id?api-version=2020-09-01 {"host": "certd.sock"}
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - !!! parameter "id" has an invalid value
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - !!! caused by: not found
sept. 05 11:01:21 myself aziot-certd[46406]: 2022-09-05T09:01:21Z [INFO] - --> 400 {"content-type": "application/json"}
sept. 05 11:01:21 myself aziot-keyd[46384]: 2022-09-05T09:01:21Z [INFO] - <-- POST /keypair?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "56"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /parameters/algorithm?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /parameters/rsa-modulus?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /parameters/rsa-exponent?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /parameters/algorithm?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /parameters/rsa-modulus?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /parameters/rsa-exponent?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:22 myself aziot-keyd[46384]: 2022-09-05T09:01:22Z [INFO] - <-- POST /encrypt?api-version=2021-05-01 {"content-length": "355", "content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:23 myself aziot-certd[46406]: 2022-09-05T09:01:23Z [INFO] - <-- POST /certificates?api-version=2020-09-01 {"content-type": "application/json", "host": "certd.sock", "content-length": "951"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - <-- GET /keypair/est-id?api-version=2021-05-01 {"host": "keyd.sock"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - <-- POST /parameters/algorithm?api-version=2021-05-01 {"content-length": "244", "content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - <-- POST /parameters/rsa-modulus?api-version=2021-05-01 {"content-length": "244", "content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - <-- POST /parameters/rsa-exponent?api-version=2021-05-01 {"content-length": "244", "content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:23 myself aziot-keyd[46384]: 2022-09-05T09:01:23Z [INFO] - <-- POST /encrypt?api-version=2021-05-01 {"content-length": "632", "content-type": "application/json"}
sept. 05 11:01:24 myself aziot-keyd[46384]: 2022-09-05T09:01:24Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:24 myself aziot-keyd[46384]: 2022-09-05T09:01:24Z [INFO] - <-- POST /encrypt?api-version=2021-05-01 {"content-length": "632", "content-type": "application/json"}
sept. 05 11:01:25 myself aziot-keyd[46384]: 2022-09-05T09:01:25Z [INFO] - --> 200 {"content-type": "application/json"}
sept. 05 11:01:25 myself aziot-certd[46406]: 2022-09-05T09:01:25Z [ERR!] - !!! internal error
sept. 05 11:01:25 myself aziot-certd[46406]: 2022-09-05T09:01:25Z [ERR!] - !!! caused by: could not create cert
sept. 05 11:01:25 myself aziot-certd[46406]: 2022-09-05T09:01:25Z [ERR!] - !!! caused by: EST endpoint did not return successful response: 400 Bad Request b"{\"errors\":[{\"code\":\"invalid_input\",\"message\":\"Please provide value for subject.organization_unit\"}]}"
sept. 05 11:01:25 myself aziot-certd[46406]: 2022-09-05T09:01:25Z [INFO] - --> 500 {"content-type": "application/json"}
sept. 05 11:01:25 myself aziot-identityd[46367]: 2022-09-05T09:01:25Z [ERR!] - Failed to provision with IoT Hub, and no valid device backup was found: internal error
sept. 05 11:01:25 myself aziot-identityd[46367]: 2022-09-05T09:01:25Z [ERR!] - service encountered an error
sept. 05 11:01:25 myself aziot-identityd[46367]: 2022-09-05T09:01:25Z [ERR!] - caused by: internal error
sept. 05 11:01:25 myself aziot-identityd[46367]: 2022-09-05T09:01:25Z [ERR!] - caused by: could not create certificate
sept. 05 11:01:25 myself aziot-identityd[46367]: 2022-09-05T09:01:25Z [ERR!] - caused by: internal error
sept. 05 11:01:25 myself aziot-identityd[46367]: 2022-09-05T09:01:25Z [ERR!] -    0: <unknown>
sept. 05 11:01:25 myself aziot-identityd[46367]:    1: <unknown>
sept. 05 11:01:25 myself systemd[1]: aziot-identityd.service: Main process exited, code=exited, status=1/FAILURE
sept. 05 11:01:25 myself systemd[1]: aziot-identityd.service: Failed with result 'exit-code'.

As you can see, the EST server rejected the request because of a missing value for subject.organization_unit.

My first question would be: is there a way to look into what the CSR really contains ?

To check if the problem could come from the EST end-point, I first created a CSR manually:

sudo -Hu aziotks OPENSSL_CONF=/var/secrets/pkcs11_openssl.conf \
  openssl req -new -sha256 -subj "/CN=my-device/OU=my-device-ou" \
    -engine pkcs11 -keyform engine -key "pkcs11:token=device;object=device-rsa-pair?pin-value=1234"  \
    -outform PEM -out device_cert.csr.pem -nodes
sudo -Hu aziotks openssl req -in temp/device_cert.csr.pem -noout -text | head -n 4

which gave:

Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: CN = my-device, OU = my-device-ou

and then sent the CSR to the EST end-point:

sudo -Hu aziotks OPENSSL_CONF=/var/secrets/pkcs11_openssl.conf \
  curl --output temp/device_cert.p7 \
  --location --request POST "$digicert_est_op_url" \
  --header "Content-Type: application/pkcs10" \
  --header "Content-Transfer-Encoding: base64" \
  --data-binary "@temp/device_cert.csr.pem" \
  --cacert "/var/secrets/cacert.crt.pem"  \
  --cert "/var/secrets/birth_cert.public.pem" \
  --key "$pkcs11_birth_key_slot"

# convert device cert from p7 to pem format
sudo openssl base64 -d -in temp/device_cert.p7 | \
 openssl pkcs7 -inform DER -outform PEM -print_certs | \
 openssl x509 -out temp/device_cert.public.pem

# mv device cert to the proper place
sudo mv temp/device_cert.public.pem /var/secrets/
sudo chown aziotks:aziotks /var/secrets/device_cert.*

sudo -Hu aziotks openssl x509 -in /var/secrets/device_cert.public.pem -text | head -n 6

one can see that the certificate could be issued:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            32:3a:5f:57:f3:48:cf:8b:ab:5f:ad:af:8e:44:4a:b2:d0:fe:8b:4e
        Signature Algorithm: sha256WithRSAEncryption

What I also tried is to use the Certificate Service to send the CSR, after having created the needed
config files in certd and keyd.
With that files configured, I could submit the CSR with this command:

sudo curl --unix-socket /run/aziot/certd.sock http://localhost/certificates?api-version=2020-09-01  \
  -H "content-type: application/json" \
  --data "$(jq -cn --arg 'certId' 'device-cert' --arg 'csr' "$(cat temp/device_cert.csr.pem)" '{"certId": $certId, "csr": $csr}')" | \
  jq -r '.pem' > temp/device_cert.public.pem

and got a valid certificate too. From this success I concluded that the problem is not sending the CSR but creating the CSR.

So my actual conclusion is: there is a problem with the CSR that the Identity Service is generating, and sending to the EST end-point

I would really appreciate any help to better diagnose the problem.

The text was updated successfully, but these errors were encountered:

arsing · 2022-09-06T02:39:00Z

As you can see, the EST server rejected the request because of a missing value for subject.organization_unit.

Yes, it's a known bug. We've received the same bug report at Azure/iotedge#6579 previously.

My first question would be: is there a way to look into what the CSR really contains ?

It's not logged anywhere by certd if that's what you're asking. You could compile your own debug build and step through with a debugger. Or you could spin up a tiny netcat server and set it as the EST endpoint.

bemol38 · 2022-09-06T09:02:13Z

Thank you @arsing. You answered my questions.
To my understanding, to build a correct CSR, the identityd service would first need to access the full subject, if any. Either the subject would need to be written to the identityd 00-super.toml too, or it should be retrievable from identityd through an api call to certd, with a new api function (get_cert_subject ?), since the subject is available in the certd 00-super.toml.
And the create_csr function would need to be rewritten too...
And maybe more ...

jlian · 2022-09-14T21:42:45Z

@bemol38 we've triaged the bug and are working on a fix. Might be slotted for a slightly later release as we line up some other fixes and security patches. How urgent is this for you?

bemol38 · 2022-09-15T09:18:22Z

Hi @jlian
First, thank you for working on a fix.
I would be very happy to be able to test this fix before end of october.

jlian · 2022-09-15T15:20:37Z

Would you be able to have DigiCert configure the EST endpoint to not require organizational_unit (OU) field in the meantime?

bemol38 · 2022-09-16T11:22:16Z

Hi @jlian,
To answer your question, yes, we are using as a temporary solution an EST end-point which does not require the OU field.

jlian · 2022-11-02T21:34:04Z

Hey folks, the change is merged (thanks @onalante-msft).

Ideally, we could have you try it before we take it for the release:

Download the patched binary here https://github.com/Azure/iot-identity-service/actions/runs/3364093065
And then manually install following these steps https://azure.github.io/iot-identity-service/installation.html

@bemol38 do you think you could give it a try this week?

bemol38 · 2022-11-03T09:37:52Z

Hi @jlian
I gave a try this morning, and it worked.
aziot service is able to build a CSR containing the OU field, as set in config.toml, and to send the CSR to our Digitcert EST Server, which now accepts it, and returns a certificate containing the OU field. The certificate lands in /var/lib/aziot/certd/certs, as expected. With that certificate, the automated provisioning is executed, the OU field is checked with a custom allocation policy and the device is registered to the IoTHub. So far so good :-)
(What I did not test already is the certificate renewal.)

Thanks a lot for your valued assistance !

jlian · 2023-02-01T20:36:35Z

Identity service 1.4.2 is available and includes this fix

pebneter · 2023-02-01T20:44:04Z

Thanks for fixing this!

arsing added the bug Something isn't working label Sep 6, 2022

jlian self-assigned this Sep 14, 2022

jlian mentioned this issue Oct 5, 2022

Identity provisioning fails to add subject DN to CSR using EST and x509 authentication when using identity certificate Azure/iotedge#6579

Closed

jlian closed this as completed Feb 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to get a certificate from EST server during x509 Provisioning when CSR subject contains additional fields (not only CN) #455

Failed to get a certificate from EST server during x509 Provisioning when CSR subject contains additional fields (not only CN) #455

bemol38 commented Sep 5, 2022

arsing commented Sep 6, 2022

bemol38 commented Sep 6, 2022

jlian commented Sep 14, 2022

bemol38 commented Sep 15, 2022

jlian commented Sep 15, 2022

bemol38 commented Sep 16, 2022

jlian commented Nov 2, 2022

bemol38 commented Nov 3, 2022 •

edited

Loading

jlian commented Feb 1, 2023

pebneter commented Feb 1, 2023

Failed to get a certificate from EST server during x509 Provisioning when CSR subject contains additional fields (not only CN) #455

Failed to get a certificate from EST server during x509 Provisioning when CSR subject contains additional fields (not only CN) #455

Comments

bemol38 commented Sep 5, 2022

arsing commented Sep 6, 2022

bemol38 commented Sep 6, 2022

jlian commented Sep 14, 2022

bemol38 commented Sep 15, 2022

jlian commented Sep 15, 2022

bemol38 commented Sep 16, 2022

jlian commented Nov 2, 2022

bemol38 commented Nov 3, 2022 • edited Loading

jlian commented Feb 1, 2023

pebneter commented Feb 1, 2023

bemol38 commented Nov 3, 2022 •

edited

Loading