integrate external device inventory proposal draft

[leoluKL]

first version.
Akri can get more device related information beyond discovery.
Suggest Akri agent talking with an external device inventory POD/service

[leoluKL]

@kate-goldenring , my bad the previous PR wrongly includes non-relevant commits. Recreate and reply your comments as below.

Is it the Agent or Discovery Handlers that will reach out to the device registry?
RE: I guess Agent does periodically run multiple discovery handlers(usb,onvif,OPC) according to Akri config, am I right? If so, I think it should be Agent to talk with external device registry instead of each discovery handler talking directly to device registry. Discovery handlers just do necessary action being defined in device protocols. I am not familiar with current discovery handler implementation detail. I suggest we can split the discovery procedure into two parts. For onvif, the first discovery part is to find host ip/MAC, and second discovery part is to extract the video stream URI. Between part 1 and part 2, Agent interferes in the middle to query device onvif credential and instruct part 2 to succeed.

Would be good to add a section defining the API in detail, specifically the contents of the query
RE: I will fill in details after we are fine with the basic ideas: Agent fetch additional device info, and embed them into customer resource Akri instance.

how will the device inventory service know to trust the Akri Agent and vice versa?
RE: My initial thought on security is just by K8s cluster admin to define network policy to allow Akri agent ingress/egress with the external device inventory POD. I guess you mean stricter way such as mutual authentication. I think from the point of view that this is a IoT Solution in a relatively safe OT on-premise environment so as initial step maybe k8s cluster network policy is good enough? Safer mutual comm (through an identity platform such as Azure AD) could be later. More importantly I think is that we should have a device management system first, that is essential for IoT solution which is my interest :)

Why not make the connection with the device inventory part of the initial discovery process in the discovery handlers? So, instances are created fully formed and all information is grabbed at once?
RE: I am align with you that once Akri instance is created it is with full device information. I am not sure if this question is still about if Akri agent or the discovery handler should talk to external device inventory. As my previous answer, I think Agent is in better place to talk to device inventory. Each discovery handler should just follow protocol (onvif, OPC. etc.) to do necessary tasks so in the future anyone who wants to contribute another protocol discovery handler, he/she just follow the protocol document but does not need to know any knowledge about talking with device inventory system.

instead of Akri hosting the service, maybe it would be better to add another endpoint to the inventory service that should also be exposed to kubernetes. This matches akris approach to date of query for devices rather than being told about them.
RE: I totally agree and I thought about the same thing as well. If Akri agent can periodically "scan" device inventory system to "find" new device just like it does onvif or OPC scan. It is best as the logic flow will be same.

but if we want the updateDevice endpoint as well, this may make sense
RE: yes, the ingress request to Akri Agent is still needed in some use cases, such as update device info.

I wonder if the /addDevice and /updateDevice services can be bundled together under some device registry extension to Akri
RE: same comments as previous two questions. We can work on this detail later after we align on the basic ideas first.

what changes to the DiscoveryHandler interface are needed to support credentials
RE: I do not dig into code level detail yet. I hope my previous imagination on splitting discover process into two parts is useful.

how to move credentials around securely
RE: Is this question about credentials of devices? Send them to broker pod as env variables will do, or not?

how new device credentials are passed to running brokers. I wonder if there needs to be some way for the device registry to pass "device registry access creds" that can be given to brokers to query for new creds when they expire but maybe this opens up the registry too much and defeats the point of rotating creds
RE: That is my suggestion of the ingress request to Akri agent, for example "/updateDevice". Akri agent should be the only one allowed to talk to device inventory. Broker pod should not be allowed to do so, as my current suggestion.

We might need to create a new threat model around this to understand the flow of trust and creds.
RE: yes, I wish to dig further on security topics. My current thought is presented in previous answers.

[leoluKL]

@kate-goldenring , I did not know device discovery handler is not embedded in Akri agent but separate POD. The proposal should clearly define the flow with discovery handler together. I update the diagram. The discovery handler is added in. To minimize the change request, I suggest a new gRPC service in Akri agent for discovery handler to talk with agent during the discovery procedure. This is necessary for discovering credential protected Onvif camera.
If you think this proposal is worth a try, may I volunteer to try developing the suggested change?

[kate-goldenring]

@kate-goldenring , I did not know device discovery handler is not embedded in Akri agent but separate POD. The proposal should clearly define the flow with discovery handler together. I update the diagram. The discovery handler is added in. To minimize the change request, I suggest a new gRPC service in Akri agent for discovery handler to talk with agent during the discovery procedure. This is necessary for discovering credential protected Onvif camera. If you think this proposal is worth a try, may I volunteer to try developing the suggested change?

@leolumicrosoft originally the discovery handlers were embedded, but we soon moved them behind a gRPC interface so that not all discovery protocols needed to be embedded in the agent. You can still conditionally compile them into the agent. I'd recommend skimming through our architecture documentation if you havent already. This diagram lays out all of akris components, this walks through the information flow, and this section on resource discovery talks about the discovery handler interface. It may make more sense to extend this interface rather than make a new one.

[leoluKL]

@kate-goldenring , yes, I already changed the proposal based on the links you shared, please review the proposal again. With keeping the current structure (discovery handler talking with Agent with gRPC call), I suggest adding a new gRPC service:RequestDeviceInfo in Agent that device handler may call it to fetch more device information through Agent. It does not deprecate current device handler. For example, current Onvif handler will still work for non-protected camera. Optionally, it is also possible to enhance current onvif handler to support Onvif authentication, by calling RequestDeviceInfo during the discovery procedure. That is drawn in proposal. I will create a PR to show the suggested code work.

[leoluKL]

A quick summary of my current progress and things I think of before submit my PR:

1. I have implemented the additional RPC function for discovery handler to call and get additional device information from an external REST POST API endpoint (which is supposed to be developer to deploy based on the actual device inventory system). This RPC function is optional, that means all the current discovery handler still works with the change.

2. I create a demo device inventory image that can be deployed to cluster for test purpose. It provide the query device endpoint and it does even have a simple web page portal for simple device data record add/delete/update operation.
   

3. I do not change current discovery configuration yaml format, instead developer can insert a optional field "query_device_http" in the disocvery detail part. If it presents, discovery handler will try to query device info from external REST endpoint.

4. To be able to re-create Akrii after device information record being added/updated/deleted, each discovery iteration not only simply compare discovered device basic info, but it will also compare the external device info, any discrepancy will trigger discovery handler report full list of devices to agent also.

5. Additional device information from external device inventory system is put into akrii .spec.brokerProperties. They can be further transported to broker pod like the current discovery device detail as environment variables in broker pod.

This PR wont not request importing additional cargo library. Current discovery handlers will still work.

Some thoughts:
I find the current onvif discovery only verify camera WSDL endpoint availability by checking its system date time, but not processing further such as check device capability, video streaming format, framerate and so on. It does not do authentication since protected onvif is not supported yet. With integration to external device inventory, it is possible to implement a more comprehensive Onvif discovery handler. I will leave it to future plan. For now, I assume broker pod developer will do those steps.

I will think how to enhance device query efficiency, right now I have to query all devices in every discovery iteration to pick up updated device info. It will be more efficient if there is a way to directly notify agent some devices' external information changes.

[kate-goldenring]

Can this proposal be built up to discuss/detail the proposed implementation: project-akri/akri#520. Specifically, detail the defined services and payloads and discuss the decision making around it.