Implicit registration of devices connected via gateways

[mhemmeter]

As of today all devices that connect to Hono via a gateway are supposed to be explicitly registered in the device registry with the via flag pointing to the gateway device. In this setup only gateway devices are authenticated and not the devices themselves. In order to simplify the management of devices in such a setup an implicit registration of devices behind a gateway would be beneficial form my perspective.

At first glance from security perspective an implicit registration should be possible as after successful authentication of the gateway device a trust relation has been established.

For example such a feature would help in case devices connect via Bluetooth to a gateway that then communicates with Hono. Due to the features in Bluetooth devices may show up in a deployment dynamically and it may not be possible to "pre-register" the devices. It would be easier in a such a dynamic scenario if Hono registers these devices implicitly upon connection instead requiring the gateway to invoke the Device Registry Management API for explicit registration.

[mhemmeter]

Remark: An interesting question is if and how an implicit registration can work in a setup where Eclipse Hono is used in combination with Eclipse Ditto (see Eclipse IoT Packages). In such a setup devices must not only be registered in the Eclipse Hono device registry but also as a Thing in Ditto.

[geglock]

Regarding the combination with Eclipse Ditto (via Eclipse IoT Packages or somehow else), a proposal could be to emit some kind of "welcome" message to the north-bound application for each implicit registration.
If such a message could be defined freely per tenant including placeholder replacement, then it could be a "Thing Create" message - see example here: https://www.eclipse.org/ditto/protocol-examples-creatething.html
This way each implicitly registered device on Hono level would also trigger the creation of a respective Thing on Ditto level.

[sophokles73]

I understand the use case and I also see the benefit of being able to register such devices on the fly. Devices being registered in this way will never be able to connect directly to a protocol adapter, though, because we would not register any credentials for them. This is consistent with the fact that the gateway device authenticates to the adapter and then acts on behalf of the edge device(s).
The biggest challenge in this context will be the management of the (edge) device identifiers.

I can imagine several approaches:

1. The gateway generates an identifier for the edge device to be registered and uses it when uploading data on behalf of the edge device. The protocol adapter and registry implementation determine that the device doesn't exist (yet) and then provision the device on the fly.
2. The gateway uses a distinct resource on the protocol adapter to register an edge device. The adapter and the registry implementation generate an identifier, register the device on the fly for the gateway and return the newly created identifier, which the gateway can then (locally) associate with the edge device.

It would also be possible to let the gateway use the gateway specific resource for uploading data on behalf of a new edge device and include a marker (header/query param etc) to indicate to the adapter that this data is for a new edge device which should be provisioned on the fly. However, it might be difficult to return the generated device identifier to the gateway in this case. In MQTT for example there is no response message which could be used to convey the information.

Regarding the additional provisioning of the device in Ditto, we should be able to use the same mechanism that we already use for devices that are being auto-provisioned based on a client certificate. See #1296 for a discussion of options. As part of that discussion we explicitly decided against emitting an event in that case for the time being. In any case, I would consider it an (optional) feature of a device registry implementation.

[sophokles73]

@calohmn @dejanb @ctron WDYT?

[avgustinmm]

If the device will have different id (chosen by gateway, adapter or registry) then it will be nice if the device has option to provide its own/internal id (if exists) in order to be incorporated in the generated id.
This way user could somehow link the device id in hono with the own device id.
If this hono id generation follows a pattern (e.g. <gateway_id>:<own_device_id>) then event this link could be made automatically be tools..

[calohmn]

FMPOV, the first approach about (the one where uploading an event/telemetry message for an unknown device triggers creation of the device) seems to be the one we should be aiming for. This would avoid having to define a separate new resource for each adapter and would also be a very intuitive solution.

I imagine, the right for a tenant to automatically create devices would have to be explicitly defined for such tenants, so that not every tenant is allowed to do that.
I could also imagine requiring an extra createIfMissing parameter for the event/telemetry requests to explicitly enable automatic device creation if needed.

Open questions:

• I guess there could be the scenario of a device being used with multiple gateways. First the device is connected with gateway A and gateway A triggers creation of the device in Hono. Then the device connects to gateway B. When gateway B wants to send an event/telemetry message on behalf of the device, that would normally get rejected because gateway A had this device be created with its own id in the device's via property. If gateway B has the same right to automatically create devices in Hono and also uses a createIfMissing parameter when sending the event/telemetry message, the protocol adapter could trigger the modification of the device entry to then also contain gateway B in the device's via list.
• Gateway groups: Support an extra request parameter with a gateway group to let it be used in the created device in a viaGroups property instead of the via property with the gateway id?
• Should there be support for letting automatically created devices also be removed again from the device registry, triggered by some kind of request to the protocol adapter? Or automatically created devices get some lifespan after which they are deleted. If the device is still needed after that period, it can just get created again the same way it got created before.

[sophokles73]

I imagine, the right for a tenant to automatically create devices would have to be explicitly defined for such tenants, so that not every tenant is allowed to do that.

FMPOV this should even require explicit authorization at the gateway level.

I guess there could be the scenario of a device being used with multiple gateways. First the device is connected with gateway A and gateway A triggers creation of the device in Hono. Then the device connects to gateway B.

I guess the desired behavior depends on whether a physical device should always be represented by a single logical device ID in Hono. If the gateway generates an ID for the device then gateway A might generate a different ID than gateway B. Following @avgustinmm 's comment above, we would always end up with different device IDs for the same physical device, depending on which gateway the device is connected via. Maybe @avgustinmm and @mhemmeter can comment on whether this would be acceptable for the envisioned used cases and what those use cases are in the first place?

Should there be support for letting automatically created devices also be removed again from the device registry.

If we create device IDs scoped to the gateway IDs then I'd rather go for the limited life span approach instead of relying on the fact that a gateway would (eventually) unregister a device ...

[e-grigorov]

Usually, the device network provides enough information for the device identifier. For example, in Blueooth, it can be the device address. Of course, the different gateway vendors can use different identifiers.

The main point here is that the gateway knows the device and can recognize it with a given identifier. It'll be useful to reuse this identifier(or similar one) in the Hone device registry. In this way, the device can be easily found.

If we create device IDs scoped to the gateway IDs then I'd rather go for the limited life span approach instead of relying on the fact that a gateway would (eventually) unregister a device ...

From the user point of view, these devices are not so different. They just cannot speak Hono protocol but they are used as any other device. Such limited life will require a periodic "keep me" message and will complicate the provisioning.

[sophokles73]

For the sake of simplicity I suggest to leave the roaming use case out of this issue for the time being. Instead, let's focus on how a gateway can register a device on the fly. Consequently, unregistering such a device should also be of no concern right now.
What I can imagine for Hono is the following:

1. A new device is being connected to a gateway that is already registered in Hono.
2. The gateway publishes a telemetry message or an event on behalf of the new device to one of Hono's protocol adapters.
3. The protocol adapter tries to determine the registration status of the device and sends a corresponding request to the Device Registration service.
4. The Device Registration service implementation fails to assert the device's registration because the device i snot registered (yet).
5. The Device Registration service checks if the gateway is authorized to register device for its tenant on the fly. This might be done by means of a property in the gateway's registration info. If the gateway is not authorized, then a 404 status is returned to the adapter and the adapter rejects the message published by the gateway. Otherwise, the registration service creates a record for the new device using the device ID provided by the gateway and sends a response to the adapter that contains a 200 code and contains a property in the defaults which indicates that the device has been registered on-the-fly, e.g. hono_registration_status=new.
6. The protocol adapter creates the downstream AMQP message from the gateway's published data and includes the hono_registration_status property in the downstream message. The adapter then forwards the AMQP message downstream.
7. A downstream application can now determine that the device has been newly created in Hono and can act accordingly. For example, Ditto could use this flag to trigger creation of a new Thing for the device before processing the message payload.

This mechanism should work regardless of the payload of the message published by the gateway. However, the gateway might also publish a specific event in order to indicate the device's capabilities as discussed in #1618. In that case Ditto could use the information from the payload to create a typed Thing that reflects the capabilities of the device already. This, however, is completely optional.

@calohmn @mhemmeter @avgustinmm @e-grigorov @geglock @fkaltner WDYT?

[mhemmeter]

The approach makes sense to me. From a requirements perspective we can exclude the roaming use case and add it later on (if necessary at all). Supporting this new feature in MQTT Protocol Adapter has highest priority for me (compared to the other protocols supported by Eclipse Hono).

[fkaltner]

I think this is doable. Can you assign this issue to me?

[thjaeckle]

I wonder if this hono_registration_status property should ever be transported via a QoS 0 telemetry message - if this message never reaches the downstream application, a corresponding thing will e.g. never be created in the downstream connected Ditto.

So FMPOV for telemetry messages there should be a separate QoS 1 event generated (ideally published before the causing telemetry message is published) in order to ensure that it eventually reaches a downstream connected consumer.
Or, alternatively, the telemetry message could be upgraded to a QoS 1 event message with message persistence, if this makes sense?

If the message from the device already was an event message, the information could be "piggy-backed" in the same QoS 1 event message via the mentioned property.

[fkaltner]

Wouldn't Ditto favor the "specific event" case Kai outlined above:

However, the gateway might also publish a specific event in order to indicate the device's capabilities as discussed in #1618. In that case Ditto could use the information from the payload to create a typed Thing that reflects the capabilities of the device already.

?

But in general you are right: for telemetry sent with QoS 0 there's a risk that the downstream application may miss that message. However, to my understanding, telemetry messages are forwarded by Hono having QoS 1 always (see: https://github.com/eclipse/hono/blob/cba0221c43ec2b93bfb436d438d8542f81f61022/client/src/main/java/org/eclipse/hono/client/impl/TelemetrySenderImpl.java#L109).
So I think the case you described shouldn't happen.

[thjaeckle]

Wouldn't Ditto favor the "specific event" case Kai outlined above:

Ah, the gateway would have to publish that event? Sure, that would also work.
I just thought that when Hono already "knows" that a new device connected via a gateway was created on-the-fly, an event could be created and published by Hono itself.

However, to my understanding, telemetry messages are forwarded by Hono having QoS 1 always ...
So I think the case you described shouldn't happen.

I don't think so. The sender link uses QoS 1 for telemetry messages as well, yes. However, there is no re-sending of negatively settled messages by the downstream consumer. Re-Sending does not happen for telemetry messages in Hono, even not for QoS 1 telemetry, as far as I know.
Such telemetry messages would have to be persisted in the broker I guess to be delivered with actual QoS 1 semantics.

[fkaltner]

Ah, the gateway would have to publish that event? Sure, that would also work.
Yes, that was my understanding. This made sense to me, since only the gateway knows what capabilities the device has.

I am not sure about the details about re-delivery - @calohmn can you help out?

[calohmn]

The sender link uses QoS 1 for telemetry messages as well, yes. However, there is no re-sending of negatively settled messages by the downstream consumer. Re-Sending does not happen for telemetry messages in Hono, even not for QoS 1 telemetry, as far as I know.

Yes. I agree that in case of a telemetry message we should send a separate event message with the hono_registration_status property then (before forwarding the telemetry message), in order to make sure that the message with that property gets brokered and will also get delivered if the downstream consumer is temporarily unavailable for example.

[fkaltner]

I am not sure why we would still need the hono_registration_status property then: how about sending a specific event in that case as well? Then we would not have two different approaches and a downstream application could always listen for those specific events in order to get notified about auto-provisioned devices.

Also this approach still has some edge cases: even if we send an event when a telemetry message of a not yet provisioned devices comes in and we wait for the event being settled we could lose the telemetry message if the application crashes at that very point of time.

[calohmn]

I am not sure why we would still need the hono_registration_status property then: how about sending a specific event in that case as well?

I think a specific event would be characterized by the very fact that it has that hono_registration_status property. That's also how the TimeUntilDisconnectNotification event messages are set up (sent out for example when a MQTT device subscribes for commands) - they are (usually) event messages containing a ttd property.

In that regard we could indeed follow the "piggy-back" approach of just adding that property if the message already is an event message. I think if the northbound application rather wants separate event messages with that status information or if it needs more information, then the gateways should be set up so that they publish specific events (#1618) in such a case.

Also this approach still has some edge cases: even if we send an event when a telemetry message of a not yet provisioned devices comes in and we wait for the event being settled we could lose the telemetry message if the application crashes at that very point of time.

But the delay in publishing the telemetry message by waiting for the disposition update for the event message first wouldn't make it more likely that the telemetry message gets lost, would it? The telemetry might get lost anyway and it's probably also not the last telemetry message of that device.

[thjaeckle]

It would be very helpful if the gateway ID could also be included as part of the message which indicates that a new device was registered.
That way downstream applications (like Ditto) could create a relation of an edge-device (or thing) to its gateway-device as well.

Maybe as additional defined property? What do you think?

[fkaltner]

Sounds reasonable and is easy to implement. So I guess there's no harm on Hono's side 😄 We could name the property hono_registration_via.

[sophokles73]

If we want to include the gateway identifier then FMPOV it should simply be gateway_id in analogy to the device_id property. In fact, we could/should include this property in every message published by a gateway on behalf of a device...

[sophokles73]

So, it seems like we include two properties in the downstream message:

1. gateway_id which contains the identifier of the gateway that acted on behalf of the device (if applicable) and
2. hono_registration_status which contains value NEW if the device has been provisioned on the fly by a gateway.