Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement API for Device Groups #3157

Merged
merged 104 commits into from
Dec 20, 2023
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
477b6cd
Add migration for DeviceGroups
Steve-Mcl Dec 2, 2023
8581dff
Add DeviceGroups model
Steve-Mcl Dec 2, 2023
a825fbe
Add App Device Groups API
Steve-Mcl Dec 2, 2023
518138d
DeviceGroups API tests
Steve-Mcl Dec 2, 2023
9a8e72e
Add createApplicationDeviceGroup support method
Steve-Mcl Dec 2, 2023
ef462b2
Add DeviceGroup db view
Steve-Mcl Dec 3, 2023
b423241
re-order DeviceGroup index position
Steve-Mcl Dec 3, 2023
be47ffc
add DeviceGroup controller
Steve-Mcl Dec 3, 2023
9420f14
delete erronous import
Steve-Mcl Dec 3, 2023
8b61c24
Update TestModelFactory to use description instead
Steve-Mcl Dec 3, 2023
207f619
load DeviceGroup controller
Steve-Mcl Dec 4, 2023
7254305
fix failing postgres tests (byId sub-clause)
Steve-Mcl Dec 4, 2023
97ae431
improve test coverage
Steve-Mcl Dec 4, 2023
dbab7ef
Test device group pagination and 404 handling
Steve-Mcl Dec 4, 2023
8084693
Update forge/db/models/DeviceGroup.js
Steve-Mcl Dec 4, 2023
bee3280
extract literals to single common literal
Steve-Mcl Dec 4, 2023
20e7bfb
correct clause for device include
Steve-Mcl Dec 4, 2023
f854a42
Cascade deletion of Application to DeviceGroup
Steve-Mcl Dec 4, 2023
70047f4
Enable schemas
Steve-Mcl Dec 4, 2023
46fa018
remove helpful stuff
Steve-Mcl Dec 4, 2023
765a826
Add frontend APIs
Steve-Mcl Dec 8, 2023
94902b1
add graphics/icons
Steve-Mcl Dec 8, 2023
65a4724
add device group management pages
Steve-Mcl Dec 8, 2023
2abfd62
hyphenate devicegroup
Steve-Mcl Dec 8, 2023
9ee7c23
Let the model validate fields and bubble up
Steve-Mcl Dec 8, 2023
d9d7a5a
corrections to device group view schemas
Steve-Mcl Dec 8, 2023
177cce0
fix double blank line lint err
Steve-Mcl Dec 8, 2023
311e523
Merge branch '2997-device-groups-backend' into 2997-device-group-fron…
Steve-Mcl Dec 8, 2023
687a9e0
updates for devicegroup hyphenation
Steve-Mcl Dec 8, 2023
f6f48cd
include device.type for devicegroup manage tables
Steve-Mcl Dec 8, 2023
57672c2
include Device Group in Device for group management
Steve-Mcl Dec 8, 2023
262f00b
Merge branch '2997-device-groups-backend' into 2997-device-group-fron…
Steve-Mcl Dec 8, 2023
6d001bf
include deviceGroupSummary for group management
Steve-Mcl Dec 8, 2023
1c7a521
Merge branch '2997-device-groups-backend' into 2997-device-group-fron…
Steve-Mcl Dec 8, 2023
bca23aa
improve table layout
Steve-Mcl Dec 8, 2023
51ab25f
use deviceGroup for filtering group management
Steve-Mcl Dec 8, 2023
d34b22f
update concepts with Device Groups info
Steve-Mcl Dec 11, 2023
d3a826b
add application device groups FE API tests
Steve-Mcl Dec 11, 2023
65016ee
add missing FE test `getDeviceGroups`
Steve-Mcl Dec 11, 2023
493ae0c
Change doc tag from "Applications" to "Application Device Group"
Steve-Mcl Dec 15, 2023
3cbb137
Move Device Groups tests to own file
Steve-Mcl Dec 15, 2023
66667ca
Merge branch 'main' into 2997-device-groups-backend
Steve-Mcl Dec 15, 2023
6eda38a
Merge branch '2997-device-groups-backend' into 2997-device-group-fron…
Steve-Mcl Dec 15, 2023
fdefa11
move to Enterprise tier
Steve-Mcl Dec 15, 2023
b91634b
Merge branch '2997-device-groups-backend' into 2997-device-group-fron…
Steve-Mcl Dec 15, 2023
9197dae
move Device Groups to enterprise tier
Steve-Mcl Dec 15, 2023
e1a2cbd
Make device group "devices" default page
Steve-Mcl Dec 15, 2023
eb5c684
Only show available devices in "edit mode"
Steve-Mcl Dec 15, 2023
820dcb7
dont show check box col in non-edit mode
Steve-Mcl Dec 15, 2023
bafd32c
remove redundant code clean up
Steve-Mcl Dec 16, 2023
f72f943
table header oddity on tab change - fix
Steve-Mcl Dec 16, 2023
a3e8bfa
ensure transaction object is passed to updates
Steve-Mcl Dec 17, 2023
4642167
ensure group icon color correctly inherits style
Steve-Mcl Dec 18, 2023
2b4f26a
Create through table for device groups in pipeline
Steve-Mcl Dec 18, 2023
3a41862
backend plumbing for pipeline device groups
Steve-Mcl Dec 18, 2023
2135610
setup frontend apis for pipeline device groups
Steve-Mcl Dec 18, 2023
ff9f211
support info/warning/success in the StatusBadge for generalised use
Steve-Mcl Dec 18, 2023
2713390
implement front end for pipeline device groups
Steve-Mcl Dec 18, 2023
0f9aad2
update concepts docs for pipeline device groups
Steve-Mcl Dec 18, 2023
ec6e13e
correction to hero title when creating new pipeline
Steve-Mcl Dec 18, 2023
fd23685
fix some texts not displaying when playing a pipeline
Steve-Mcl Dec 18, 2023
fde19b5
fix polling - correction to event hookups
Steve-Mcl Dec 18, 2023
ff3cfdd
fix icon fill color inheritance
Steve-Mcl Dec 18, 2023
5ffe19f
revert temp debugging options
Steve-Mcl Dec 18, 2023
31c42c9
refactor pipeline stage updates and common error handling
Steve-Mcl Dec 18, 2023
617dd44
add model validator to ensure device groups in same app
Steve-Mcl Dec 18, 2023
d7fc2fd
update unit tests
Steve-Mcl Dec 18, 2023
2f94f04
check device-groups feature enabled
Steve-Mcl Dec 18, 2023
2694276
fix UI test - differentiate "Device" from "Device Group"
Steve-Mcl Dec 18, 2023
666440c
Add device group frontend e2e tests
Steve-Mcl Dec 18, 2023
e78ded9
Merge branch '2997-device-group-frontend' into 2989-assign-device-gro…
Steve-Mcl Dec 18, 2023
36a69f1
revert unnecessary change (not needed)
Steve-Mcl Dec 18, 2023
15c2772
really fix e2e test - only click the button without "Group"
Steve-Mcl Dec 18, 2023
ca2a0de
remove extraneous it.only
Steve-Mcl Dec 18, 2023
2d14a40
remove duplicate tests left over from splitting out to own file
Steve-Mcl Dec 18, 2023
14e4e8a
Merge branch '2997-device-groups-backend' into 2997-device-group-fron…
Steve-Mcl Dec 18, 2023
0a642a7
Merge branch '2997-device-group-frontend' into 2989-assign-device-gro…
Steve-Mcl Dec 18, 2023
fb419c4
walk pipeline stages, dont assume id order is correct!
Steve-Mcl Dec 19, 2023
a95acfa
fix intermittent PG test fails
Steve-Mcl Dec 19, 2023
ccb05c0
refactor sortStages to model static fn & add unit tests
Steve-Mcl Dec 19, 2023
af7d8f2
fix PG test bugs - ensure we wait the correct thing
Steve-Mcl Dec 19, 2023
a76f343
Merge branch 'main' into 2997-device-groups-backend
Steve-Mcl Dec 19, 2023
97c3ed0
Merge branch '2997-device-groups-backend' into 2997-device-group-fron…
Steve-Mcl Dec 19, 2023
bef6d99
Merge branch '2997-device-group-frontend' into 2989-assign-device-gro…
Steve-Mcl Dec 19, 2023
1d05674
Add device group feature flag in TeamType
knolleary Dec 19, 2023
d307835
Enable deviceGroups feature on test team type
knolleary Dec 19, 2023
8ca537e
Add device group events to audit log
Steve-Mcl Dec 19, 2023
7f7939f
Ensure deviceGroup team flag is passed up
knolleary Dec 19, 2023
d524e15
Clarify team-loading behaviour
knolleary Dec 20, 2023
bd9c7c2
Add loading graphic to device groups page
knolleary Dec 20, 2023
2ced020
Ensure team is loaded when loading application
knolleary Dec 20, 2023
390291b
Merge pull request #3235 from FlowFuse/2989-device-group-team-tier-en…
knolleary Dec 20, 2023
386a54e
Add audit logger for application.pipeline.stage-deployed
Steve-Mcl Dec 20, 2023
cf597c8
Add call to audit log for pipeline stage deployment
Steve-Mcl Dec 20, 2023
295f68d
add ui element
Steve-Mcl Dec 20, 2023
0a1110a
add all missing tests for team.application.* audit loggers
Steve-Mcl Dec 20, 2023
ef99669
Merge pull request #3239 from FlowFuse/2989-add-missing-audit-logger-…
Pezmc Dec 20, 2023
5b11555
Merge pull request #3229 from FlowFuse/2989-assign-device-group-to-pi…
knolleary Dec 20, 2023
137e2b8
Merge pull request #3192 from FlowFuse/2997-device-group-frontend
knolleary Dec 20, 2023
49aa7fa
Merge branch 'main' into 2997-device-groups-backend
knolleary Dec 20, 2023
f60cf2f
Update and rename 20231202-01-add-device-devicegroup.js to 20231220-0…
knolleary Dec 20, 2023
c29af2d
Move DeviceGroup controller function to DG Controller
knolleary Dec 20, 2023
8af208d
Merge branch '2997-device-groups-backend' into 2997-device-group-audi…
Steve-Mcl Dec 20, 2023
4796eca
Merge pull request #3236 from FlowFuse/2997-device-group-auditloggers
Pezmc Dec 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
231 changes: 231 additions & 0 deletions forge/db/controllers/DeviceGroup.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,231 @@
const { Op } = require('sequelize')

const { ControllerError } = require('../../lib/errors')
class DeviceGroupMembershipValidationError extends ControllerError {
/**
* @param {string} code
* @param {string} message
* @param {number} statusCode
* @param {Object} options
*/
constructor (code, message, statusCode, options) {
super(code, message, statusCode, options)
this.name = 'DeviceGroupMembershipValidationError'
}
}

module.exports = {

/**
* Create a Device Group
* @param {import("../../forge").ForgeApplication} app The application object
* @param {string} name The name of the Device Group
* @param {Object} options
* @param {Object} [options.application] The application this Device Group will belong to
* @param {string} [options.description] The description of the Device Group
* @returns {Promise<Object>} The created Device Group
*/
createDeviceGroup: async function (app, name, { application = null, description } = {}) {
// Create a Device Group that devices can be linked to
// * name is required
// * application, description are optional
// * FUTURE: colors (background, border, text) and icon will optional

if (typeof name !== 'string' || name.length === 0) {
throw new Error('Tag name is required')
Steve-Mcl marked this conversation as resolved.
Show resolved Hide resolved
}
return await app.db.models.DeviceGroup.create({
name,
description,
ApplicationId: application?.id
})
},

updateDeviceGroup: async function (app, deviceGroup, { name = undefined, description = undefined } = {}) {
// * deviceGroup is required.
// * name, description, color are optional
if (!deviceGroup) {
throw new Error('DeviceGroup is required')
}
let changed = false
if (typeof name !== 'undefined') {
if (typeof name !== 'string' || name.length === 0) {
throw new Error('Tag name is required')
}
deviceGroup.name = name
changed = true
}
if (typeof description !== 'undefined') {
deviceGroup.description = description
changed = true
}
if (changed) {
await deviceGroup.save()
await deviceGroup.reload()
}
return deviceGroup
},

updateDeviceGroupMembership: async function (app, deviceGroup, { addDevices, removeDevices, setDevices } = {}) {
// * deviceGroup is required. The object must be a Sequelize model instance and must include the Devices
// * addDevices, removeDevices, setDevices are optional
// * if setDevices is provided, this will be used to set the devices assigned to the group, removing any devices that are not in the set
// * if addDevices is provided, these devices will be added to the group
// * if removeDevices is provided, these devices will be removed from the group
// if a device appears in both addDevices and removeDevices, it will be removed from the group (remove occurs after add)
if (!setDevices && !addDevices && !removeDevices) {
return // nothing to do
}
if (!deviceGroup || typeof deviceGroup !== 'object') {
throw new Error('DeviceGroup is required')
}
let actualRemoveDevices = []
let actualAddDevices = []
const currentMembers = await deviceGroup.getDevices()
// from this point on, all IDs need to be numeric (convert as needed)
const currentMemberIds = deviceListToIds(currentMembers, app.db.models.Device.decodeHashid)
setDevices = setDevices && deviceListToIds(setDevices, app.db.models.Device.decodeHashid)
addDevices = addDevices && deviceListToIds(addDevices, app.db.models.Device.decodeHashid)
removeDevices = removeDevices && deviceListToIds(removeDevices, app.db.models.Device.decodeHashid)

// setDevices is an atomic operation, it will replace the current list of devices with the specified list
if (typeof setDevices !== 'undefined') {
// create a list of devices that are currently assigned to the group, minus the devices in the set, these are the ones to remove
actualRemoveDevices = currentMemberIds.filter(d => !setDevices.includes(d))
// create a list of devices that are in the set, minus the devices that are currently assigned to the group, these are the ones to add
actualAddDevices = setDevices.filter(d => !currentMemberIds.includes(d))
} else {
if (typeof removeDevices !== 'undefined') {
actualRemoveDevices = currentMemberIds.filter(d => removeDevices.includes(d))
}
if (typeof addDevices !== 'undefined') {
actualAddDevices = addDevices.filter(d => !currentMemberIds.includes(d))
}
}

// wrap the dual operation in a transaction to avoid inconsistent state
const t = await app.db.sequelize.transaction()
try {
// add devices
if (actualAddDevices.length > 0) {
await this.assignDevicesToGroup(app, deviceGroup, actualAddDevices)
}
// remove devices
if (actualRemoveDevices.length > 0) {
await this.removeDevicesFromGroup(app, deviceGroup, actualRemoveDevices)
}
// commit the transaction
await t.commit()
} catch (err) {
// Rollback transaction if any errors were encountered
await t.rollback()
// if the error is a DeviceGroupMembershipValidationError, rethrow it
if (err instanceof DeviceGroupMembershipValidationError) {
throw err
}
// otherwise, throw a friendly error message along with the original error
throw new Error(`Failed to update device group membership: ${err.message}`)
}
},

assignDevicesToGroup: async function (app, deviceGroup, deviceList) {
const deviceIds = await validateDeviceList(app, deviceGroup, deviceList, null)
await app.db.models.Device.update({ DeviceGroupId: deviceGroup.id }, { where: { id: deviceIds.addList } })
},

/**
* Remove 1 or more devices from the specified DeviceGroup
* @param {*} app The application object
* @param {*} deviceGroupId The device group id
* @param {*} deviceList A list of devices to remove from the group
*/
removeDevicesFromGroup: async function (app, deviceGroup, deviceList) {
const deviceIds = await validateDeviceList(app, deviceGroup, null, deviceList)
// null every device.DeviceGroupId row in device table where the id === deviceGroupId and device.id is in the deviceList
await app.db.models.Device.update({ DeviceGroupId: null }, { where: { id: deviceIds.removeList, DeviceGroupId: deviceGroup.id } })
}
}

/**
* Convert a list of devices to a list of device ids
* @param {Object[]|String[]|Number[]} deviceList List of devices to convert to ids
* @param {Function} decoderFn The decoder function to use on hashes
* @returns {Number[]} Array of device IDs
*/
function deviceListToIds (deviceList, decoderFn) {
// Convert a list of devices (object|id|hash) to a list of device ids
const ids = deviceList?.map(device => {
let id = device
if (typeof device === 'string') {
[id] = decoderFn(device)
} else if (typeof device === 'object') {
id = device.id
}
return id
})
return ids
}

/**
* Verify devices are suitable for the specified group:
*
* * All devices in the list must either have DeviceGroupId===null or DeviceGroupId===deviceGroupId
* * All devices in the list must belong to the same Application as the DeviceGroup
* * All devices in the list must belong to the same Team as the DeviceGroup
* @param {*} app The application object
* @param {*} deviceGroupId The device group id
* @param {*} deviceList A list of devices to verify
*/
async function validateDeviceList (app, deviceGroup, addList, removeList) {
// check to ensure all devices in deviceList are not assigned to any group before commencing
// Assign 1 or more devices to a DeviceGroup
if (!deviceGroup || typeof deviceGroup !== 'object') {
throw new Error('DeviceGroup is required')
}

// reload with the Application association if not already loaded
if (!deviceGroup.Application) {
await deviceGroup.reload({ include: [{ model: app.db.models.Application }] })
}

const teamId = deviceGroup.Application.TeamId
if (!teamId) {
throw new Error('DeviceGroup must belong to an Application that belongs to a Team')
}

const deviceIds = {
addList: addList && deviceListToIds(addList, app.db.models.Device.decodeHashid),
removeList: removeList && deviceListToIds(removeList, app.db.models.Device.decodeHashid)
}
const deviceGroupId = deviceGroup.id
if (deviceIds.addList) {
const okCount = await app.db.models.Device.count({
where: {
id: deviceIds.addList,
[Op.or]: [
{ DeviceGroupId: null },
{ DeviceGroupId: deviceGroupId }
],
ApplicationId: deviceGroup.ApplicationId,
TeamId: teamId
}
})
if (okCount !== deviceIds.addList.length) {
throw new DeviceGroupMembershipValidationError('invalid_input', 'One or more devices cannot be added to the group', 400)
}
}
if (deviceIds.removeList) {
const okCount = await app.db.models.Device.count({
where: {
id: deviceIds.removeList,
DeviceGroupId: deviceGroupId,
ApplicationId: deviceGroup.ApplicationId,
TeamId: teamId
}
})
if (okCount !== deviceIds.removeList.length) {
throw new DeviceGroupMembershipValidationError('invalid_input', 'One or more devices cannot be removed from the group', 400)
}
}
return deviceIds
}
1 change: 1 addition & 0 deletions forge/db/controllers/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ const modelTypes = [
'ProjectTemplate',
'ProjectSnapshot',
'Device',
'DeviceGroup',
'BrokerClient',
'StorageCredentials',
'StorageFlows',
Expand Down
62 changes: 62 additions & 0 deletions forge/db/migrations/20231202-01-add-device-devicegroup.js
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs renaming as we have more recent migrations already merged.

Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
/* eslint-disable no-unused-vars */
/**
* Add DeviceGroups table that has an FK association with Applications and Devices
*/

const { Sequelize, QueryInterface } = require('sequelize')

module.exports = {
/**
* upgrade database
* @param {QueryInterface} context QueryInterface
*/
up: async (context) => {
await context.createTable('DeviceGroups', {
id: {
allowNull: false,
autoIncrement: true,
primaryKey: true,
type: Sequelize.INTEGER
},
name: {
type: Sequelize.STRING(255),
allowNull: false
},
description: {
type: Sequelize.TEXT
},
createdAt: {
allowNull: false,
type: Sequelize.DATE
},
updatedAt: {
allowNull: false,
type: Sequelize.DATE
},
ApplicationId: {
type: Sequelize.INTEGER,
references: {
model: 'Applications',
key: 'id'
},
onUpdate: 'CASCADE',
onDelete: 'SET NULL'
}
})

// add a column to Devices table that references DeviceGroups
await context.addColumn('Devices', 'DeviceGroupId', {
type: Sequelize.INTEGER,
references: {
model: 'DeviceGroups',
key: 'id'
},
onUpdate: 'CASCADE',
onDelete: 'SET NULL'
})
},

down: async (queryInterface, Sequelize) => {

}
}
1 change: 1 addition & 0 deletions forge/db/models/Application.js
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ module.exports = {
this.hasMany(M.Project)
this.hasMany(M.Project, { as: 'Instances' })
this.belongsTo(M.Team, { foreignKey: { allowNull: false } })
this.hasMany(M.DeviceGroup, { foreignKey: { allowNull: true } })
},
finders: function (M) {
return {
Expand Down
1 change: 1 addition & 0 deletions forge/db/models/Device.js
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ module.exports = {
this.belongsTo(M.ProjectSnapshot, { as: 'activeSnapshot' })
this.hasMany(M.DeviceSettings)
this.hasMany(M.ProjectSnapshot) // associate device at application level with snapshots
this.belongsTo(M.DeviceGroup, { foreignKey: { allowNull: true } }) // SEE: forge/db/models/DeviceGroup.js for the other side of this relationship
},
hooks: function (M, app) {
return {
Expand Down