Update device on group membership change #3330

Steve-Mcl · 2024-01-12T12:41:57Z

Description

Premise: The customer wishes for devices added to a group to receive the snapshot last pushed via a pipeline. Additionally, the customer wishes for the pipeline deployed snapshot to be removed from the device when it is removed from the group.

~~Adds field activePipelineStageId to DeviceGroups so that added/removed devices can query the active pipeline stage and be updated accordingly~~
Adds field targetSnapshotId to DeviceGroups so that added/removed devices can query the DeviceGroup and be updated accordingly
Updates UI to warn user of the consequences by way of a confirmation dialog (shown when user attempts to save changes to a device group)
Immediately transmits (in MQTT coms mode) an update to the entire group and any devices removed
Updated test coverage
- Adds: Add and Remove Devices get an update request
- ~~Adds: All Devices removed get an update request and clears activePipelineStageId~~
- Adds: All Devices removed get an update request and clears DeviceGroup.targetSnapshotId
Updated docs

Related Issue(s)

#3260

Checklist

I have read the contribution guidelines
Suitable unit/system level tests have been added and they pass
Documentation has been updated
- Upgrade instructions
- Configuration details
- Concepts
Changes flowforge.yml?
- Issue/PR raised on FlowFuse/helm to update ConfigMap Template
- Issue/PR raised on FlowFuse/CloudProject to update values for Staging/Production

Labels

Backport needed? -> add the backport label
Includes a DB migration? -> add the area:migration label

Add and Remove Devices get an update request All Devices removed get an update request and clears activePipelineStageId

Steve-Mcl · 2024-01-12T12:44:08Z

@ZJvandeWeg not specifically adding you as a reviewer, but I thought to tag you as docs around device-groups will be updated as per comment here: #3318 (comment)

codecov · 2024-01-12T12:49:01Z

Codecov Report

Attention: 41 lines in your changes are missing coverage. Please review.

Comparison is base (16ffc90) 40.92% compared to head (64fd77d) 41.25%.
Report is 29 commits behind head on main.

Files	Patch %	Lines
...tend/src/pages/application/DeviceGroup/devices.vue	0.00%	26 Missing ⚠️
...s/20240116-01-add-targetSnapshot-to-DeviceGroup.js	0.00%	5 Missing ⚠️
forge/ee/db/models/PipelineStageDeviceGroup.js	28.57%	5 Missing ⚠️
forge/ee/db/controllers/DeviceGroup.js	92.68%	3 Missing ⚠️
frontend/src/components/pipelines/Stage.vue	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3330      +/-   ##
==========================================
+ Coverage   40.92%   41.25%   +0.32%     
==========================================
  Files         607      608       +1     
  Lines       22762    22865     +103     
  Branches     5494     5535      +41     
==========================================
+ Hits         9315     9432     +117     
+ Misses      13447    13433      -14

Flag	Coverage Δ
backend	`78.26% <77.58%> (+0.66%)`	⬆️
frontend	`2.04% <0.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

knolleary · 2024-01-12T13:19:29Z

@Steve-Mcl think I'd like to do a huddle review of this PR with you on Monday. This may be building on choices already made, but I'd like to understand better why the device group needs the activePipelineStageId adding - we already have the PipelineStageDeviceGroup model that holds the relationships between DeviceGroup and PipelineStage which, I think, could be used instead of adding this column. I appreciate I may be missing something here - easier to step through together on Monday.

Steve-Mcl · 2024-01-12T13:25:54Z

@Steve-Mcl think I'd like to do a huddle review of this PR with you on Monday. This may be building on choices already made, but I'd like to understand better why the device group needs the activePipelineStageId adding - we already have the PipelineStageDeviceGroup model that holds the relationships between DeviceGroup and PipelineStage which, I think, could be used instead of adding this column. I appreciate I may be missing something here - easier to step through together on Monday.

Sure, NP.

For the memory banks and later discussion:

A device group can be assigned to multiple pipelines. Without activePipelineStageId there is no way of knowing which of these pipeline stage was deployed.
To permit us to indicate which pipeline stage was/is executing, (and as a nice side-effect) prevent all pipelines in the page animating as if deploying when one is operated, each PipelineStageDeviceGroup row holds the targetSnapshotId that was deployed. This is the value we need to get to understand WHICH of the multiple pipelines was executed.

knolleary · 2024-01-15T11:31:14Z

forge/db/migrations/20240110-01-add-active-pipeline-id-to-devicegroup.js

We have since merged a migration with a newer filename to main. This migration will need renaming.

done 9e88a2c

docs/user/device-groups.md

Steve-Mcl · 2024-01-16T12:26:36Z

@knolleary review ready following changes discussed. OP updated, tests updated etc.

knolleary · 2024-01-16T14:17:30Z

forge/ee/db/controllers/DeviceGroup.js

+            // check to see if the group is now empty
+            const remainingDevices = await deviceGroup.deviceCount()
+            if (remainingDevices === 0) {
+                deviceGroup.targetSnapshotId = null


Suggested change

deviceGroup.targetSnapshotId = null

deviceGroup.targetSnapshotId = null

Wondering if setting the snapshot to null when the group is emptied is the right behaviour. A user could be in the middle of changing the devices in the group (albeit doing it in multiple steps) - and clearing the snapshot mid-way through the task could be unexpected.

I don't think it is of consequence.

it just means the pipeline needs to be executed to make it re-sync.

The issue this was added for is adding devices to a spare empty group (months/years later) and suddently they fire up running old (dangerous?) snapshots.

I feel quite strongly that we DONT actually provide a checkbox [x] remove current snapshot / [x] apply current snapshot - I think it is dangerous and suspect most (other) users will not like the behaviour we are adding at all

From the ISSUE #3260 (comment):

@MarianRaphael I believe it would be prudent to consider this:

Personally, I believe this should be done via an option when adding a device to a group since you may NOT actually want this to occur immediately and without warning that "things" could start moving/happening.

Consider the scenario:

You have x devices that you need to add to a device group BUT FIRST you need to update the source snapshot
The current snapshot is NOT what needs to be deployed (because changes have not yet been made to accommodate the 10 new devices)
You CANNOT/DO NOT want to deploy the new changes to the existing devices without the other devices being online/ready to receive the new snapshot.
By simply adding a checkbox "[x] Auto Deploy current device group snapshot 'version 2.1'?" at the time of adding the device to a group can make this problem go away!

knolleary · 2024-01-16T14:23:16Z

forge/ee/db/controllers/Pipeline.js

@@ -432,6 +432,9 @@ module.exports = {
                // Update the targetSnapshot of the device group
                await targetDeviceGroup.PipelineStageDeviceGroup.update({ targetSnapshotId: sourceSnapshot.id }, { transaction })

+                // Update the targetSnapshotId on the device group
+                await targetDeviceGroup.update({ targetSnapshotId: sourceSnapshot.id }, { transaction })
+


A possibly cleaner approach would be to have a controller for DeviceGroup which provides a function to update the targetSnapshotId which takes care of updating the devices and sending the update command. As it is we've got that logic spread through this Pipeline controller. Not blocking, just a thought.

I could add a method to DeviceGroup controller but I dont want to be updating devices until the transaction is committed.

Steve-Mcl added 9 commits January 10, 2024 12:24

add activePipelineStageId to DeviceGroups

414ec39

Update device group members and active pipeline stage

b88a4d7

ensure activeSnapshotId is cleared when running default SS

4be5476

Add confirm dialog for updating device group members

c92b1db

add tests

c6b5099

Add and Remove Devices get an update request All Devices removed get an update request and clears activePipelineStageId

ensure targetSnapshotId and activePipelineStageId are set

e640b03

Merge branch 'main' into 3260-update-device-on-group-membership-change

0e20240

Improve confirmation dialog

7581043

Update docs

5e044c4

Steve-Mcl added the area:migration Involves a database migration label Jan 12, 2024

Steve-Mcl requested a review from knolleary January 12, 2024 13:07

knolleary reviewed Jan 15, 2024

View reviewed changes

update migration date

9e88a2c

ZJvandeWeg reviewed Jan 15, 2024

View reviewed changes

docs/user/device-groups.md Outdated Show resolved Hide resolved

Link to related sections

769e49c

ZJvandeWeg reviewed Jan 15, 2024

View reviewed changes

docs/user/device-groups.md Show resolved Hide resolved

Steve-Mcl added 5 commits January 16, 2024 09:07

Merge branch 'main' into 3260-update-device-on-group-membership-change

d547ac5

change migration to targetSnapshot

b471286

use targetSnapshotId instead of activePipelineStageId

0d5b51e

Use state object for reflecting status

604f48a

comment and var name improvements following rework

64fd77d

Steve-Mcl requested a review from knolleary January 16, 2024 12:26

knolleary reviewed Jan 16, 2024

View reviewed changes

knolleary self-requested a review January 16, 2024 15:29

knolleary approved these changes Jan 17, 2024

View reviewed changes

knolleary merged commit ab96f6a into main Jan 17, 2024

fix error toast

a975bbe

Steve-Mcl mentioned this pull request Jan 17, 2024

Use Alert.emit instead of $toast #3352

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update device on group membership change #3330

Update device on group membership change #3330

Steve-Mcl commented Jan 12, 2024 •

edited

Steve-Mcl commented Jan 12, 2024

codecov bot commented Jan 12, 2024 •

edited

knolleary commented Jan 12, 2024

Steve-Mcl commented Jan 12, 2024

knolleary Jan 15, 2024

Steve-Mcl Jan 15, 2024

Steve-Mcl commented Jan 16, 2024

knolleary Jan 16, 2024

Steve-Mcl Jan 16, 2024

knolleary Jan 16, 2024

Steve-Mcl Jan 16, 2024

	deviceGroup.targetSnapshotId = null
	deviceGroup.targetSnapshotId = null

Update device on group membership change #3330

Update device on group membership change #3330

Conversation

Steve-Mcl commented Jan 12, 2024 • edited

Description

Related Issue(s)

Checklist

Labels

Steve-Mcl commented Jan 12, 2024

codecov bot commented Jan 12, 2024 • edited

Codecov Report

knolleary commented Jan 12, 2024

Steve-Mcl commented Jan 12, 2024

knolleary Jan 15, 2024

Choose a reason for hiding this comment

Steve-Mcl Jan 15, 2024

Choose a reason for hiding this comment

Steve-Mcl commented Jan 16, 2024

knolleary Jan 16, 2024

Choose a reason for hiding this comment

Steve-Mcl Jan 16, 2024

Choose a reason for hiding this comment

knolleary Jan 16, 2024

Choose a reason for hiding this comment

Steve-Mcl Jan 16, 2024

Choose a reason for hiding this comment

Steve-Mcl commented Jan 12, 2024 •

edited

codecov bot commented Jan 12, 2024 •

edited