Provide means to record manual off-line device updates into hawkBit's history

[stormc]

When having to manually update a device, e.g., for failure-recovery from a bad update, it would be nice to have a means to record this manual update into hawkBit's history so that it can serve as the single source of truth regarding the updates done to a device.

Currently, one can simulate the effect by creating an according softwaremodule/distributionset and "installing" that update on a device simulator resembling the real device in terms of configuration. However, this is a bit tedious and may mess up the history, e.g., in case the real device has performed an update in the meantime.

[kaizimmerm]

I like the idea. We could use this also. At the end of the day it should possible to post completed actions into the history. Would also be help full to teach hawkBit the initial installation state or even an executed factory reset.

[StBurcher]

Short summary of our idea: Using PutConfigDat with an extended DdiConfigData. So a client is able to send configuration information to hawkbit.
The DdiRootcontroller calls updateControllerAttributes and put all Key, Value pairs (Version and Name) into the TargetInfo.
So far so good. But where is the event that something has changed? In order to react to the change-

[kaizimmerm]

https://github.com/eclipse/hawkbit/blob/master/hawkbit-repository/hawkbit-repository-api/src/main/java/org/eclipse/hawkbit/repository/eventbus/event/TargetUpdatedEvent.java will be send by the repository.

After you have received the event based on the attributes you can create a "completed update action" on the target. However, that function does not exist yet.

I guess https://github.com/eclipse/hawkbit/blob/master/hawkbit-repository/hawkbit-repository-jpa/src/main/java/org/eclipse/hawkbit/repository/jpa/JpaDeploymentManagement.java could be extended to create such a thing, i.e. create a complete action and set installed DS. (keep in mind that if you intend to contribute this into hawkBit we need not only the repository extension but also the Mgmt API extension as well).

Last step for the required functionality is to create a small service that listens to the event, searches for a fitting DS and uses the new repository function to update the action history and installed DS.

if you intend to contribute this last functionality as well we have to think about a hawkBit plugin mechanism. Maybe @michahirsch has a good idea how to structure this. Spring auto configuration should be a good basis for such a plugin mechanism.

[michahirsch]

as @kaizimmerm mentioned we are thinking about a place in the hawkBit repository like plugins, addons or extensions which can be contributed and hosted in the hawkBit Github repository and plugged in using the Spring-Configuration mechanism. We'll come up with a concept and example but I would definitely split up the functionality of complete action and installed DS and the functionality to automatically do it based on an update event into different maven-modules.

[StBurcher]

@kaizimmerm and @michahirsch
After revisiting the events. I would suggest to send Installed DS Details via putConfigData as part of the DdiConfigDate map. Adding them with targetInfo.getControllerAttributes().putAll(data); into the TargetInfo and send an TargetInfoUpdate Event.
Next Step creating a new TargetEventHandler Class in the same Package as EventMerger. If a targetUpdateInfoEvent appears, the Class looks after a key i.e. DSInformation= "update". In this case the specific keys will extract from the attributs. The DS information will be used to find the DS in the DB. If it does not exist an error will be thrown. Otherwise the set will connected to the target.

[singh-himanshu]

@kaizimmerm and @michahirsch
we are coming up with an implementation based on comments from @kaizimmerm and @StBurcher that could also address #446 to some extent. what i wanted to check is if we have some concept available for plug-in mechanism already? we can try to align the implementation with the concept if it is there.

[a-sayyed]

@singh-himanshu recently I have been looking at this problem, and seeing that you are already working on it, perhaps I could join-in and help?

[singh-himanshu]

@a-sayyed, as an initial implementation, instead of leveraging the PutConfigData and TargetUpdatedEvent as suggested by @StBurcher and @kaizimmerm , we currently provide an additional API to directly create a software module, a distribution set, assign it to specified controllers and then close the action.
We have tried to keep the dependencies to minimal by implementing the feature as an extension that can be disabled through application properties.
There are few more things to be done such as being able to upload multiple artifacts to the server while creating the software module. Idea is that we not only just record the updates, but also have ability to get the artifacts, so that when replayed, we can recreate the device's current state.

[kaizimmerm]

Hi everyone,

I have a client request that needs the manual insert of a finished action through Management API/UI. So I would teach repository and management API to do this (which is basically the following part of what I described above):

I guess https://github.com/eclipse/hawkbit/blob/master/hawkbit-repository/hawkbit-repository-jpa/src/main/java/org/eclipse/hawkbit/repository/jpa/JpaDeploymentManagement.java could be extended to create such a thing, i.e. create a complete action and set installed DS. (keep in mind that if you intend to contribute this into hawkBit we need not only the repository extension but also the Mgmt API extension as well).

However, has anyone done this already and can a pull request be expected soon? Otherwise I would start looking into this.

Kai

[singh-himanshu]

We did work out an implementation by creating an extension project to add an API that can take a multi-part request in case we want to upload the artefacts as well to hawkBit. Approach is essentially similar to suggested above, i.e. extend JpaDeploymentManagement. Following is an excerpt from developer comments. Although I'll have to check whether we are ready to push it upstream already.

This feature provides a new REST API which can be invoked to send the details of software updates performed offline on a single or a set of devices. The response is almost similar to that of creation of a new Distribution Set.

Mechanism:

1. Retrieves the list of controller Ids from the input.
2. Checks if the targets exist for corresponding controllers. If not then throw appropriate exception.
3. Create the software modules. Upload artefacts if present, else update only metadata with filename and corresponding URL.
4. Create a Distribution Set for the given module types. The new distribution set name is generated in the >format DS_ and the version is always 1.0.0.
5. For each controller, create an action by assigning the distribution set.
6. Mark all the actions created above as complete.

API : {tenant}/controller/v1/updateOfflineTarget

Request:

curl -X POST
http://127.0.0.1:8080/TENANT_ID/controller/v1/updateOfflineTarget -H
'accept: application/json' -H 'content-type: multipart/form-data; -F
'offlineUpdateInfo={
  "softwareModules":[
    {
      "name":"offlineModule",
      "description":"operating system updated offline",
      "artifactData":[
        {
          "filename":"service.bin",
          "md5Hash":"a10f3b3bbdf5536550f6ac76e5a26db3",
          "sha1Hash":"dd31d32801d571c833295a568373b92d81ba2e15",
          "href":"https://FILESERVER/software/os/updates/service.bin",
          "version":"1.0"
        } ],
        "type":"os",
        "version":"1.0.0"
        } ],
  "migrationStepRequired":true,
  "controllerIds":["controller1"]
}'

Response:

{
  "createdBy": "admin",
  "createdAt": 1495121741558,
  "lastModifiedBy": "admin",
  "lastModifiedAt": 1495121741558,
  "name":
"DS_DEFAULT_1495121741474_06d0fbf3-b575-48b7-b5c7-8776bfbb9b2b",
  "version": "1.0.0",
  "modules": [
    {
      "createdBy": "admin",
      "createdAt": 1495121741438,
      "lastModifiedBy": "admin",
      "lastModifiedAt": 1495121741466,
      "name": "offlineModule",
      "description": "operating system updated offline",
      "version": "1.0.0",
      "type": "os",
      "_links": {
        "self": {
          "href": "http://127.0.0.1:8080/rest/v1/softwaremodules/1"
        },
        "type": {
          "href": "http://127.0.0.1:8080/rest/v1/softwaremoduletypes/1"
        }
      },
      "id": 1
    } ],
  "migrationStepRequired": true,
  "type": "os",
  "complete": true,
  "_links": {
    "self": {
      "href": "http://127.0.0.1:8080/rest/v1/distributionsets/1"
    },
    "type": {
      "href": "http://127.0.0.1:8080/rest/v1/distributionsettypes/2"
    },
    "metadata": {
      "href":
"http://127.0.0.1:8080/rest/v1/distributionsets/1/metadata?offset=0&limit=50{&sort,q}",
      "templated": true
    }
  },
  "id": 1
}

[kaizimmerm]

Uhh, thats much bigger than what I need :)

Well, I am looking forward to see that pull request. I would assume that 5/6 are a feature on its own (and pretty much what I am looking for). The DDI extension is another one.

One question on that one already. Are you sure that its needed to be done for group of devices hawkBits DDI security is single controller based, i.e. there is no security mechanism to authorise device on DDI to handle stuff for other devices.

So if you need to do it for multiple devices would management API (with the extension to create "completed" actions) not be the right tool as well for you? There is the alternative of course that you go back to single device handling and keep this a DDI feature if its needed there.

Anyway, if you could provide the DeploymentManagement extension to create offline updates as a pull request I could move on from there with the Management API extension and you could contribute the DDI extension later. Smaller pull requests make the whole review process a lot faster as well ;) As alternative I could take care of the DeploymentManagement extension and you integrate this into your contribution.

Let me know what you think.

[singh-himanshu]

Well, I am looking forward to see that pull request. I would assume that 5/6 are a feature on its own (and pretty much what I am looking for). The DDI extension is another one.

Yes the implementation is a bit big as its done as a new extension project that provides a single point to (a) upload the artefacts to hawkBit, (b) create software modules using them, (c) find a 'best' DistributionSetType, and (d) create a completed action for all controllers.
We should be able to push it soon so that you can take a look and recommend how we may slice/dice it.

One question on that one already. Are you sure that its needed to be done for group of devices hawkBits DDI security is single controller based, i.e. there is no security mechanism to authorise device on DDI to handle stuff for other devices.

You are right, actually we created a management API and not a DDI. Guess the URL mapping was left to be updated. Later if we need to support self registration from devices, we can add a DDI API.

[kaizimmerm]

Please check this pull request: #563 which should do the trick.