| title | authors | reviewers | creation-date | last-updated | status | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Edge Device Management |
|
|
2021-03-10 00:00:00 UTC |
2021-03-10 00:00:00 UTC |
provisional |
- Managing Edge Devices using EdgeX Foundry
Refer to the OpenYurt Glossary and the EdgeX Foundry Glossary.
This proposal introduces an approach leverages existing edge computing platforms, like EdgeX Foundry, and uses Kubernetes custom resources to abstract edge devices. When adapting Kubernetes to the edge computing environments, existing solutions either change the system architecture (k3s wrap together the controlplane and the kubelet) or modify core components (Kubelet in KubeEdge) significantly. However, inspiring by the Unix philosophy, "Do one thing and do it well", we believe that Kubernetes should focus on managing computing resources while edge devices management can be done by adopting existing edge computing platforms. Therefore, we define several generic custom resource definitions(CRD) that act as the mediator between OpenYurt and the edge platform. Any existing edge platforms can be integrated into the OpenYurt by implementing custom controllers for these CRDs. In addition, these CRDs allow users to manage edge devices in a declarative way, which provides users with a Kubernetes-native experience.
Extending the Kubernetes to the edge is not a new topic. However, to support all varieties of edge devices, existing Kubernetes-based edge frameworks have to develop dedicated adaptors for each category of the device from scratch and change the original Kubernetes architecture significantly, which entails great development efforts with the loss of some upstream K8S features. Instead, we are inspired by the Unix philosophy, i.e., “Do one thing, and do it well,” and believe that a mature edge IoT platform should do the device management. To that end, we integrate EdgeX into the OpenYurt, which supports a full spectrum of devices and supports all upstream K8S features.
- To design a new custom resource definition(CRD), DeviceProfile, to represent different categories of devices
- To design a new CRD, Device, that represents physical edge devices
- To design a new CRD, DeviceService, that defines the way of how to connect to a specific device
- Using EdgeX Foundry as an example platform, implement controllers that support device management using DeviceProfile, Device, DeviceService and EdgeX Foundry
- To support automatically setting up of the EdgeX Foundry on the OpenYurt
- To support declarative device state modification, i.e., modifying the device's properties by changing fields of the device CRs
Non-goals are limited to the scope of this proposal. These features may evolve in the future.
- To implement a EdgeX Foundry DeviceService for any specific protocol
- To support data transmission between edge devices and external services
- To support edge data processing
- As a vendor, I would like to connect a category of devices into OpenYurt.
- As an end-user, I would like to define how to connect a device, which belongs to a supported DeviceProfile, into the OpenYurt.
- As an end-user, I would like to connect a new device, which belongs to a supported DeviceProfile, to the OpenYurt.
- As an end-user, I would like to modify the states of devices by changing the values of device properties defined in corresponding Device CRs.
- As an end-user, I would like to disconnect a device by deleting the corresponding Device CR.
We define three new custom resource definitions(CRD) to manage devices on the edge. They are DeviceProfile, Device, and DeviceService. The DeviceProfile defines a type of devices using same kind of protocol, which includes some generic information like the manufacturer's name, the device description, and the device model. DeviceProfile also defines what kind of resources (e.g., temperature, humidity) this type of device provided and how to read/write these resources. The DeviceService defines the way of how to connect a device to the OpenYurt, like the URL of the device. The DeviceService can not exist alone. Every DeviceService must associate with a DeviceProfile. The Device gives the detailed definition of a specific device, like which DeviceProfile it belongs to and which DeviceService it used to connect to the system.
Followings are definitions of DeviceProfile CRD and its related Golang structs. This CRD will be defined by vendors, and the corresponding CRs will be created by device controller when it detected a new device service created.
type DeviceProfile struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec DeviceProfileSpec `json:"spec,omitempty"`
Status DeviceProfileStatus `json:"status,omitempty"`
}
// DeviceProfileSpec defines the desired state of DeviceProfile
type DeviceProfileSpec struct {
// Description of the Device
Description string `json:"description,omitempty"`
// Manufacturer of the device
Manufacturer string `json:"manufacturer,omitempty"`
// Model of the device
Model string `json:"model,omitempty"`
// Available DeviceResources of the profile
DeviceResources []DeviceResource `json:"deviceResources,omitempty"`
// Available CoreCommands of the profile
CoreCommands []Command `json:"coreCommands,omitempty"`
}
// DeviceProfileStatus defines the observed state of DeviceProfile
type DeviceProfileStatus struct {
// ProfileId is the Id assigned by the backend platform, e.g., EdgeX Foundry
ProfileId string `json:"id,omitempty"`
// Condition indicates the condition of the DeviceProfile.
Condition DeviceProfileCondition `json:"addedToEdgeX,omitempty"`
}
type DeviceProfileCondition string
const (
// The DeviceProfile is unavailable, i.e., has not registered on
// the backend platform yet
Unavailable DeviceProfileCondition = "Unavailable"
// The DeviceProfile is available, i.e., successfully registered
//on the backend platform
Available DeviceProfileCondition = "Available"
)
// DeviceResource is the resource/data that
type DeviceResource struct {
Description string `json:"description"`
Name string `json:"name"`
Property ProfileProperty `json:"property"`
}
// ProfileProperty defines the property details of the DeviceResource
type ProfileProperty struct {
Value PropertyValue `json:"value"`
Units string `json:"units,omitempty"`
}
// PropertyValue defines the value format of the property
type PropertyValue struct {
// ValueDescriptor Type of property after transformations
Type PropertyValueType `json:"type,omitempty"`
// Read/Write Permissions set for this property
Mutable bool `json:"mutable,omitempty"`
// Minimum value that can be get/set from this property
Minimum string `json:"minimum,omitempty"`
// Maximum value that can be get/set from this property
Maximum string `json:"maximum,omitempty"`
// Default value set to this property if no argument is passed
DefaultValue string `json:"defaultValue,omitempty"`
}
type PropertyValueType string
const (
ValueTypeBool PropertyValueType = "Bool"
ValueTypeString PropertyValueType = "String"
ValueTypeUint8 PropertyValueType = "Uint8"
ValueTypeUint16 PropertyValueType = "Uint16"
ValueTypeUint32 PropertyValueType = "Uint32"
ValueTypeUint64 PropertyValueType = "Uint64"
ValueTypeInt8 PropertyValueType = "Int8"
ValueTypeInt16 PropertyValueType = "Int16"
ValueTypeInt32 PropertyValueType = "Int32"
ValueTypeInt64 PropertyValueType = "Int64"
ValueTypeFloat32 PropertyValueType = "Float32"
ValueTypeFloat64 PropertyValueType = "Float64"
ValueTypeBinary PropertyValueType = "Binary"
ValueTypeBoolArray PropertyValueType = "BoolArray"
ValueTypeStringArray PropertyValueType = "StringArray"
ValueTypeUint8Array PropertyValueType = "Uint8Array"
ValueTypeUint16Array PropertyValueType = "Uint16Array"
ValueTypeUint32Array PropertyValueType = "Uint32Array"
ValueTypeUint64Array PropertyValueType = "Uint64Array"
ValueTypeInt8Array PropertyValueType = "Int8Array"
ValueTypeInt16Array PropertyValueType = "Int16Array"
ValueTypeInt32Array PropertyValueType = "Int32Array"
ValueTypeInt64Array PropertyValueType = "Int64Array"
ValueTypeFloat32Array PropertyValueType = "Float32Array"
ValueTypeFloat64Array PropertyValueType = "Float64Array"
)
// Command defines the available commands for end users to control the devices
// NOTE: a Command usually corresponding to a DeviceCommand and a DeviceResource
type Command struct {
// Command name (unique on the profile)
Name string `json:"name,omitempty"`
// Get Command
Get Get `json:"get,omitempty"`
// Put Command
Put Put `json:"put,omitempty"`
}
// Put defines the details of the Put operation
type Put struct {
Path string `json:"path,omitempty"`
ParameterNames []string `json:"parameterNames,omitempty"`
}
// Get defines the details of the Get operation
type Get struct {
Path string `json:"path,omitempty"`
}Followings are the definition of the Device CRD and its related Golang structs. This CRD will be defined by vendors, while the corresponding CRs will be created by users:
// Device is the Schema for the devices API
type Device struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec DeviceSpec `json:"spec,omitempty"`
Status DeviceStatus `json:"status,omitempty"`
}
// DeviceSpec defines the desired state of Device
type DeviceSpec struct {
// Device service specific location
Location string `json:"location,omitempty"`
// Associated Device Service - One per device
Service string `json:"service"`
// Associated Device Profile - Describes the device
Profile string `json:"profile"`
// DeviceProperties defines the desired states of device properties
DeviceProperties map[string]DeviceProperty `json:"deviceProperties,omitempty"`
}
// DeviceStatus defines the observed state of Device
type DeviceStatus struct {
DeviceId string `json:"deviceId,omitempty"`
// Registered indicates whether the object has been successfully
// created on the backend platform, e.g., EdgeX Foundry
Condition DeviceCondition `json:"condition,omitempty"`
// Time (milliseconds) that the device last provided any feedback or
// responded to any request
LastConnected metav1.Time `json:"lastConnected,omitempty"`
// Time (milliseconds) that the device reported data to the core
// microservice
LastReported metav1.Time `json:"lastReported,omitempty"`
// DeviceProperties contains the observed stats of device properties
DeviceProperties map[string]DeviceProperty `json:"deviceProperties,omitempty"`
}
type DeviceCondition string
const (
Unavailable DeviceCondition = "Unavailable"
Available DeviceCondition = "Available"
Error DeviceCondition = "Error"
)
type DeviceProperty struct {
Name string `json:"name"`
Value string `json:"value"`
}Followings are definitions of DeviceService CRD and related Golang structs. This CRD will be defined by vendors, and the corresponding CRs will be created by device controller when it detected a new device service created.
// DeviceService is the Schema for the deviceservices API
type DeviceService struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec DeviceServiceSpec `json:"spec,omitempty"`
Status DeviceServiceStatus `json:"status,omitempty"`
}
// DeviceServiceSpec defines the desired state of DeviceService
type DeviceServiceSpec struct {
Description string `json:"description,omitempty"`
// address (MQTT topic, HTTP address, serial bus, etc.) for reaching the service
Addressable Addressable `json:"addressable,omitempty"`
}
// DeviceServiceStatus defines the observed state of DeviceService
type DeviceServiceStatus struct {
// ServiceId is the Id assigned by the EdgeX foundry
ServiceId string `json:"id,omitempty"`
// The condition of the service
Condition DeviceServiceCondition `json:"condition,omitempty"`
}
type DeviceServiceCondition string
const (
Unavailable DeviceServiceCondition = "Unavailable"
Available DeviceServiceCondition = "Availale"
)
type Addressable struct {
// ID is a unique identifier for the Addressable, such as a UUID
Id string `json:"id,omitempty"`
// Name is a unique name given to the Addressable
Name string `json:"name,omitempty"`
// Protocol for the address (HTTP/TCP)
Protocol string `json:"protocol,omitempty"`
// Address of the addressable
Address string `json:"address,omitempty"`
// Port for the address
Port int `json:"port,omitempty"`
// Path for callbacks
Path string `json:"path,omitempty"`
// For message bus protocols
Publisher string `json:"publisher,omitempty"`
// Topic for message bus addressables
Topic string `json:"topic,omitempty"`
}In the next release version, we will reveal the new component, DeviceManager, which will help us to manage edge devices through EdgeX Foundry. In this section, we will show how we interact with the EdgeX Foundry with DeviceManager. Though the first version of DeviceManager can only work with the EdgeX Foundry, other edge platforms should be able to integrated into OpenYurt in a very similar way.
The deployment strategy for EdgeX Foundry may vary based on the cluster topology. We will define a CRD, which will be discussed in another proposal, to control the way of deploying EdgeX services on OpenYurt. Also, we will allow users to choose if they want to set up the EdgeX Foundry automatically when converting the Kubernetes cluster to the OpenYurt cluster using the yurtctl command-line tool.
The DeviceManager includes three controllers, i.e., DeviceProfile Controller, DeviceService Controller, and the Device Controller, which act as mediators between the OpenYurt and the EdgeX Foundry and are responsible for reconciling the states of the device-related custom resources with the states of the corresponding objects on the EdgeX Foundry.
Following is the process of connecting a new device to the OpenYurt through EdgeX Foundry.
- The vendor applies the DeviceProfile CR.
- The DeviceManager creates the related ValueDescriptor on the EdgeX MetaData Service.
- The DeviceManager create the DeviceProfile on the EdgeX Metadata Service.
- The end-user applies the Device CR.
- During the runtime, command service will read available commands from the Metadata Service, translate commands and execute them.
- The end-user applies the DeviceService CR.
- The DeviceManager creates the DeviceService object on the EdgeX Metadata Service.
- The DeviceManager creates the Device object on the EdgeX Metadata Service.
- The Physical Device is connected to the OpenYurt through the device service instance (The actual microservice that talks to the physical devices depend on what protocol the device uses, the device service instance may be deployed as a pod on edge nodes or be deployed outside of the OpenYurt cluster).
The declarative model is one of the core concepts of Kubernetes. This model allows users to declare the desired states of resources/workloads, and the corresponding controllers/operators will reconcile the actual states with the desired states. We will add a new field, DeviceProperties, to the Device CRD, which will hold the device properties that can be modified declaratively. The general idea behind the DeviceProperties is that the vendor (or anyone who defines the Device CRD) will decide which DeviceResource can be changed declaratively and create a matching entry in the DeviceProperties. After the Device CR is applied to the cluster, the device controller will query the EdgeX Foundry command service, fetch the rest endpoint, and update the DeviceProperties entries.
For example, following is a DeviceProfile CR represents a type of sensor devices
apiVersion: device.openyurt.io/v1alpha1
kind: DeviceProfile
metadata:
name: color-sensor
spec:
...
deviceResources:
- name: lightcolor
description: "JSON message"
properties:
value:
{ type: "String", readWrite: "W" , mediaType : "application/json" }
units:
{ type: "String", readWrite: "R" }
...
coreCommands:
- name: lightcolor
get:
path: "/api/v1/device/{deviceId}/color"
responses:
-
code: "200"
description: "get current light color"
expectedValues: ["color"]
-
code: "503"
description: "service unavailable"
expectedValues: []
put:
path: "/api/v1/device/{deviceId}/changeColor"
responses:
-
code: "201"
description: "set the light color"
-
code: "503"
description: "service unavailable"
expectedValues: []The DeviceProfile contains one deviceResources, i.e., lightcolor, which supports both "read" and "write" operations. The profile also defines a corresponding coreCommand, which contains the REST API of the lightcolor. After this DeviceProfile is applied to the cluster, the following core command will be created on the EdgeX Foundry,
{
...
"commands": [
{
"created": <created-timestamp>,
"modified": <modifed-timestamp>,
"id": "<command-id>",
"name": "lightcolor",
"get": {
"path": "/api/v1/device/{deviceId}/color",
"responses": [
{
"code": "200",
"description": "get current light color",
"expectedValues": [
"color"
]
},
{
"code": "503",
"description": "service unavailable"
}
],
"url": "http://edgex-core-command:48082/api/v1/device/<device-id>/command/<set-command-id>"
},
"put": {
"path": "/api/v1/device/{deviceId}/changeColor",
"responses": [
{
"code": "201",
"description": "set the light color"
},
{
"code": "503",
"description": "service unavailable"
}
],
"url": "http://edgex-core-command:48082/api/v1/device/<device-id>/command/<put-command-id>"
}
}
]
}Then we can create a Device associated to the DeviceProfile as following,
apiVersion: device.openyurt.io/v1alpha1
kind: Device
metadata:
name: testsensor
spec:
...
profile: color-sensor
deviceProperties:
lightcolor:
name: lightcolor
desiredValue: greenAssume that the initial light color is blue, then after applying the Device, the device controller will try to fetch the corresponding get/put URLs from the core command service, update the testsensor.Spec.DeviceProperties[lightcolor] testsensor.Status.DeviceProperties[lightcolor], and then set the light color to green, the updated Device will look like the following,
apiVersion: device.openyurt.io/v1alpha1
kind: Device
metadata:
name: testsensor
spec:
...
profile: color-sensor
deviceProperties:
lightcolor:
name: lightcolor
setURL: http://edgex-core-command:48082/api/v1/device/<device-id>/command/<put-command-id>
desiredValue: green
status:
deviceProperties:
lightcolor:
name: lightcolor
getURL: http://edgex-core-command:48082/api/v1/device/<device-id>/command/<set-command-id>
desiredValue: greenEdge devices are usually located in the network regions that are unreachable from the cloud, while DeviceService needs to talk to edge devices directly. Therefore, we need to deploy the DeviceService on the edge node that connects to edge devices. On the other hand, when modifying the device properties, the device controller will need to send a request to the DeviceService. Hence, we will deploy a replica of the device controller in each network region. Specifically, the device controller will connect to the YurtHub, and the YurtHub will only pull the device CRDs related to connected edge devices. The system architecture is shown below,
In the first implementation, we will support the EdgeX Foundry Hanoi, and would not support upgrading/downgrading to other versions.k
- 03/15/2021: Proposed idea in an issue or community meeting.
- 03/16/2021: Present proposal at a community meeting and collect feedbacks.
- 04/13/2021: Present the revised proposal at a community meeting and collect feedbacks.
- 04/14/2021: Finalize the proposal.