DeviceStreamRequest from within a Module in IoT Edge 1.2, 1.4

[emilm]

Hello!

Before in 1.1.x I could create a DeviceClient by using the device ID and DeviceAuthenticationWithTpm for TPM devices and
DeviceAuthenticationWithX509Certificate for devices using X509.

How is this possible for TPM devices in IoT Edge 1.4 when aziot-tpmd service seems to be claiming the TPM device and is in conflict with tpm2-abrmd service? Is there another way to do Device Streams from within a module?

Thanks!

[jlian]

aziot-tpmd service seems to be claiming the TPM device and is in conflict with tpm2-abrmd service

Do you have the logs?

[emilm]

aziot-tpmd service seems to be claiming the TPM device and is in conflict with tpm2-abrmd service

Do you have the logs?

Sorry, yes. My host OS has the tpm2-abrmd service so I can use the DeviceAuthenticationWithTpm from within my module through dbus.

Log from tpm2-abrmd on my system if started after aziot-tpmd:

systemd[1]: tpm2-abrmd.service: Scheduled restart job, restart counter is at 2601.
systemd[1]: Stopped TPM2 Access Broker and Resource Management Daemon.
systemd[1]: Starting TPM2 Access Broker and Resource Management Daemon...
tpm2-abrmd[19327]: tcti_conf before: "device:/dev/tpm0"
tpm2-abrmd[19327]: tcti_conf after: "device:/dev/tpm0"
tpm2-abrmd[19327]: ERROR:tcti:../tpm2-tss-2.3.2/src/tss2-tcti/tcti-device.c:439:Tss2_Tcti_Device_Init() Failed to open device file /dev/tpm0: Device or resource busy

Log from aziot-tpmd if started after tpm2-abrmd:

aziot-tpmd[20011]: WARNING:tcti:../tpm2-tss-2.3.2/src/tss2-tcti/tctildr.c:62:tcti_from_init() TCTI init for function 0x7658465d failed with a000a
aziot-tpmd[20011]: WARNING:tcti:../tpm2-tss-2.3.2/src/tss2-tcti/tctildr.c:92:tcti_from_info() Could not initialize TCTI named: tcti-device
aziot-tpmd[20011]: ERROR:tcti:../tpm2-tss-2.3.2/src/tss2-tcti/tctildr-dl.c:150:tcti_from_file() Could not initialize TCTI file: device
aziot-tpmd[20011]: ERROR:tcti:../tpm2-tss-2.3.2/src/tss2-tcti/tctildr.c:418:Tss2_TctiLdr_Initialize_Ex() Failed to instantiate TCTI
aziot-tpmd[20011]: 2023-03-21T14:49:18Z [ERR!] - service encountered an error
aziot-tpmd[20011]: 2023-03-21T14:49:18Z [ERR!] - caused by: internal error
aziot-tpmd[20011]: 2023-03-21T14:49:18Z [ERR!] - caused by: could not initialize TPM
aziot-tpmd[20011]: 2023-03-21T14:49:18Z [ERR!] - caused by: tcti:IO failure
aziot-tpmd[20011]: 2023-03-21T14:49:18Z [ERR!] - 0:

So there’s clearly a conflict here, and I am not sure where it says that it can’t co-exist with tpm2-abrmd.
If started alone, they seem to work fine. Using DeviceAuthenticationWithTpm after this (C# device SDK) seems to crash though.

So the question is if I have to make my own implementation of DeviceAuthenticationWithTpm and use the aziot-tpmd socket directly, or if there's other means to instantiate a DeviceClient within a module in order to use Device Streams.

[bilalsellak]

Yes, it looks like both are trying to use the device file and that would cause an issue since it can be only used by one process at a time.
@drwill-ms @veyalla This is an interesting use case and wondering if you have any input on SDK usage for these situations in 1.4

[drwill-ms]

We're not supporting the device streams feature in the .NET SDK at this time. The feature is still in preview by the IoT Hub service team and we've heard no plans to GA it. It still exists in an older preview version so it is there for evaluation purposes only.

Also, TPM support is broken on Linux due to the TPM library dependency. They have fixed the bug but will not release a new package. For that reason (and others), we're dropping TPM auth support in our v2 release. However, the error above looks like a different TPM issue, which simply says the TPM was busy, which I would simply put in a retry loop with a small delay.

[emilm]

We're not supporting the device streams feature in the .NET SDK at this time. The feature is still in preview by the IoT Hub service team and we've heard no plans to GA it. It still exists in an older preview version so it is there for evaluation purposes only.

Ok, it has been in preview since forever. But is there a replacement / alternative for this? Last time I talked to a support representative I was told it would not be removed at least. It is a very useful feature for remote access, and should be a part of the offering I think!

Also, TPM support is broken on Linux due to the TPM library dependency. They have fixed the bug but will not release a new package. For that reason (and others), we're dropping TPM auth support in our v2 release. However, the error above looks like a different TPM issue, which simply says the TPM was busy, which I would simply put in a retry loop with a small delay.

What happens is that aziot-tpmd is running seemingly having a forever lasting handle on the TPM device without releasing it.
Because what happens is that my module fires up, try to use DeviceAuthenticationWithTpm, which will start tpm2-abrmd service which was not previously started on the host OS.
Then it will time out because the service fails to start because the TPM device is busy.

Could you check if the aziot-tpmd opens and closes the TPM context for each operation , or is it kept open? Can it be closed for each request with an option perhaps? It is important that your service will not claim the device exclusively because it will break other services accessing it.

[drwill-ms]

Ok, it has been in preview since forever. But is there a replacement / alternative for this?

That has been our complaint as well. Maintaining a preview SDK takes effort and the team that owns hasn't been moving forward with it as of yet.

Last time I talked to a support representative I was told it would not be removed at least.

It isn't removed from the service preview API version, but it is still technically a preview meaning it should only be used for evaluation. The feature won't appear in our non-preview SDK releases therefore it won't be officially supported.

It is a very useful feature for remote access, and should be a part of the offering I think!

I've heard from a handful of passionate customers about the feature, but for some reason for another it hasn't risen to the top of the backlog for the IoT Hub service team.

Independent of device streams, it sounds like what we have is a problem for a device and a module both trying to use the TPM to authenticate on the same machine. Is that right?

[emilm]

Independent of device streams, it sounds like what we have is a problem for a device and a module both trying to use the TPM to authenticate on the same machine. Is that right?

Yes, that is correct! My suspicion is that the aziot-tpmd is holding on to the TPM device during it's lifespan.
I am looking at the tpmd code: https://github.com/Azure/iot-identity-service/blob/main/tpm/aziot-tpmd/src/lib.rs#L135

At a quick glance of the code: My understanding is that the TPM context is passed into it and it will keep using that context. Maybe this needs to be changed to some sort of a factory opening and closing the context for each operation ?

not sure how Intel's tpm2-abrmd is but it seems like the caller makes the decision how long to claim the device.

My proposal is one or more of the following (if my assumptions are true)

• Refactor aziot-tpmd to open / close TPM device per request (but what happens when tpm2-abrmd is running , it seems to claim the tpm as well... )
• refactor aziot-tpmd to use the dbus API to communicate with tpm2-abrmd instead of directly (which is what SecurityProviderTpm does ? ) - https://github.com/microsoft/TSS.MSR/blob/master/TSS.NET/TSS.Net/LinuxTpmDevice.cs#L136
• I have to make my own SecurityProviderTpm or something that will communicate with aziot-tpmd socket, since it has essentially taken over the role for tpm2-abrmd, but I have no idea how.

That has been our complaint as well. Maintaining a preview SDK takes effort and the team that owns hasn't been moving forward with it as of yet.

Such a shame! I hope it will be put into GA. Once it's there it will be hard to move away from it!

[bilalsellak]

You can configure what TCTI module tpmd uses in your config.toml (if trying to interact with tpm2-abrmd)
[tpm]
tcti = "swtpm:port=2321"

Some additional details

iotedge/edgelet/contrib/config/linux/template.toml

Line 269 in dc5c99e

# [tpm]

[emilm]

You can configure what TCTI module tpmd uses in your config.toml (if trying to interact with tpm2-abrmd) [tpm] tcti = "swtpm:port=2321"

Some additional details

iotedge/edgelet/contrib/config/linux/template.toml

Line 269 in dc5c99e

# [tpm]

Thank you! But is this configured in the aziot-edged toml ?
The aziot-tpmd config template says "# This file intentionally left empty because aziot-tpmd has no configurable settings."

[bilalsellak]

Thank you! But is this configured in the aziot-edged toml ? The aziot-tpmd config template says "# This file intentionally left empty because aziot-tpmd has no configurable settings."

Yes, in your "super-config" found at (by default) /etc/aziot/config.toml

[emilm]

Thank you! But is this configured in the aziot-edged toml ? The aziot-tpmd config template says "# This file intentionally left empty because aziot-tpmd has no configurable settings."

Yes, in your "super-config" found at (by default) /etc/aziot/config.toml

Thank you, it worked! (after a reboot) I will produce a new OS image and test it from scratch!
So is it enough to disable the aziot-tpmd service, and nothing from iot edge will try to start it if this is in the config?

[arsing]

The aziot-tpmd config template says "# This file intentionally left empty because aziot-tpmd has no configurable settings."

We need to fix that, since it does have config now.

Thank you, it worked! (after a reboot)

Can you post the config you used to use abrmd. We should document it as an example in the template instead of, or at least before, the swtpm example.

So is it enough to disable the aziot-tpmd service, and nothing from iot edge will try to start it if this is in the config?

The aziot-tpmd service should stay disabled because it'll be started by systemd when anything requests the aziot-tpmd.socket. And to be clear, aziot-tpmd will start. The point of that config is that aziot-tpmd will talk to tpm2-abrmd instead of to /dev/tpm* directly.