Remote device access, editing and snapshotting with Auth enabled #2042

Steve-Mcl · 2023-04-25T10:48:41Z

Description

Support remote device access, editing and snapshotting with NR token Auth to prevent local access

TODO:

documentation
tests
audit log entries - open a task to capture this

Notes for debugging and comprehension

The auth method is essentially as described in the issue mermaid diagram except for the route names as described by Bens comment in the following comment n the issue
Search the code for "Enable Device Editor" to see the 12 steps of acheiving a successful tunnel
2. Each step indicates source and destination, protocol and a brief description
3. This will also aid completing documentation
I stand on the shoulders of giants - Ben. The tunnelling was first realised by Ben (thanks @hardillb) BUT adding Auth and controlling the tunnel, reigning in the many of async problems (had to add cmnd/resp support!), head twisting, torturous job to get it all working (to prevent local access but permit tunnelled access). So my apologies for the jumbled tired comments, lateness of this PR and the size of it but after working ~48h SOLID (Fri ~ Now) and debugging 401s, 503s, xhr cancelled, no response, no auto-login, wrong token, wrong auth header, missing auth headers, alternative auth methods and a thousand other problems, I am really really t...

Related PRs

FlowFuse/device-agent#77

Related Issue(s)

#1821

Checklist

I have read the contribution guidelines
Suitable unit/system level tests have been added and they pass
Documentation has been updated
- Upgrade instructions
- Configuration details
- Concepts
Changes flowforge.yml?
- Issue/PR raised on flowforge/helm to update ConfigMap Template
- Issue/PR raised on flowforge/CloudProject to update values for Staging/Production

Labels

Backport needed? -> add the backport label
Includes a DB migration? -> add the area:migration label

to get positive response to a command (ie not fire and forget) necessary for interlocking and correct representation of tunnel state before user attempts access Done in a generic way to permit future command/response opportunities Adds callback to MQTT publish and `device.sendCommand` Adds response topic to device MQTT Updates MQTT ACLs for response topic Adds `comms.device` > `sendCommandAsync()` and `sendCommandAwaitReply()`

Adds mode flag and API for setting Device Mode (autonomous/programmer) Updates device view to include mode and tunnel info Add permissions preHandlers Make the whole process more interlocked by using the new TunnelManager and utelising the new Cmnd/Resp MQTT calls Document the 12 steps to connection (please keep comments for future ref Ensure WS connections close on error (add include WS error codes) Document APIs

support head and options dont look for `projectSettings` on a tunnelled device (fixes crash) improve tracing (through comments) of the 12 steps to achieve a tunnel fix auth by switching to x-access-token

non enterprise should not show elements of this feature add Developer Mode Only adjacent to the section title flask tooltip to read "Developer Mode" re-word "Device Mode" choice dialog

Steve-Mcl · 2023-05-02T16:48:36Z

Following dev-demo live (p)review, additional pushes have been made to address ui/ux and security.

Security

URL for Admin is /device-editor/ ✅
access_token=xxxx appended to editor url ✅
FF Platform account necessary to access (browser should have session)
- I.e. only operable when logged in to FF ✅
- I.e. not operable from another browser using proxy URL or local access ✅

Enterprise Only

Elements are hidden for community edition ✅

UI / UX

Developer Options - add smaller, lighter “Developer Mode Only” text to the right ✅
Device Editor should be 3 columns ✅
- Label
- State
- Toggle Button
Create Snapshot should be 3 columns ✅
- Label
- Blank
- Create Snapshot button
Tooltip for the flask icon
- Should say “Developer Mode” ✅
Device Mode Modal
- Remove mention of “Autonomous mode” from UI - essentially, Developer mode on/off ✅

Still todo (either this or later iteration)

Timeout of enabling tunnel should revert state to show actual state
- We witnessed the device being "broken" but tunnel still showed "enabled". State should reflect condition.
- NOTE: This is slightly different now we only have a "toggle" button for enabling/disabling device editor access

knolleary · 2023-05-02T17:03:42Z

Timeout of enabling tunnel should revert state to show actual state
We witnessed the device being "broken" but tunnel still showed "enabled". State should reflect condition.
NOTE: This is slightly different now we only have a "toggle" button for enabling/disabling device editor access

I think we definitely need something in the first iteration then ensures the state doesn't get out of sync if the device is offline. Otherwise it will say the editor is enabled, but the button never comes active - without any indication what is wrong.

…into device-editor

and for licensed platform only

Steve-Mcl · 2023-05-04T13:28:26Z

New Tests passing (except the common Reached heap limit Allocation failed - JavaScript heap out of memory)

knolleary · 2023-05-09T08:22:36Z

Is there a summary of all of the new api routes this PR adds? It's quite hard to get the big picture trying to pick them all out individually and the external apis are a key part of the design.

PRs of this size and complexity should ideally summarise their key changes in the description, including db changes, api changes etc.

Given this is EE licensed functionality, some of this should be under the ee src tree - such as the DeviceTunnelManager and api/deviceEditor - so the relevant routes only get added if a license is applied.

Timeout of enabling tunnel should revert state to show actual state
We witnessed the device being "broken" but tunnel still showed "enabled". State should reflect condition.
NOTE: This is slightly different now we only have a "toggle" button for enabling/disabling device editor access

Has this been resolved in the subsequent commits?

Refactor device editor endpoints and move under ee code tree

Steve-Mcl · 2023-05-10T08:16:11Z

@knolleary

I have merged and performed local testing (approx 1hr, enable, disable, with MQTT down, with device up/down etc)
Much more solid with sendOrQueue mod to agent. (thanks for that)

fixed up button text wrapping in f82ad24

Other than that, ready for review/merge.

frontend/src/pages/device/Overview.vue

Co-authored-by: Nick O'Leary <nick.oleary@gmail.com>

docs/user/devices.md

Steve-Mcl · 2023-05-10T09:31:54Z

forge/routes/api/device.js

+        reply.send(app.db.views.ProjectSnapshot.snapshot(snapShot))
+    })
+
+    async function assignDeviceToProject (device, project) {


Suggested change

async function assignDeviceToProject (device, project) {

// #region Helpers

async function assignDeviceToProject (device, project) {

Missing #region marker

hardillb and others added 14 commits April 14, 2023 15:33

First pass at Device Editor access

3c9acc3

Merge branch 'main' into device-editor

0be8e61

Add DeviceTunnelManager class to manage start/stop

56ab954

Utelise Cmnd/Resp to tie up the async operations

ff2fdba

add "mode" autonomous/programmer to Devices table

8113580

Add snapshot support (backend)

becb83b

Add frontend code for mode, editor and snapshots

4439134

typedef improvements for type inference and checks

2e076f4

fix sending of command payload in MQTT command

6817207

bug fixes and code tidy

8e7d67a

support head and options dont look for `projectSettings` on a tunnelled device (fixes crash) improve tracing (through comments) of the 12 steps to achieve a tunnel fix auth by switching to x-access-token

pass token in url on launching editor

88f0e25

short delay b4 emit "device updated" timing fix

0ada24a

Steve-Mcl mentioned this pull request Apr 25, 2023

Remote device access, editing and snapshotting with Auth enabled FlowFuse/device-agent#77

Merged

13 tasks

Steve-Mcl linked an issue Apr 25, 2023 that may be closed by this pull request

Access the Node-RED Editor on Devices #1821

Closed

Steve-Mcl and others added 3 commits May 2, 2023 17:15

only FF authenticated sessions to access editor

2f7982c

ui/ux updates following dev-demo

2ebbf1a

non enterprise should not show elements of this feature add Developer Mode Only adjacent to the section title flask tooltip to read "Developer Mode" re-word "Device Mode" choice dialog

Merge branch 'main' into device-editor

ea688ee

Steve-Mcl added 8 commits May 3, 2023 10:02

fix layout issues since 3 column layout added

8a22fc2

fix redundant attribute causing noise in console

59bd251

fix tunnel connect fail status showing connected

2b9a693

Merge branch 'device-editor' of https://github.com/flowforge/flowforge …

2d588fe

…into device-editor

remove leftover debug console logging

cc1ec90

remove temporary debugging fastify plugin timeout

ff9e770

tidy up frontend code (more dry)

550e0db

Merge remote-tracking branch 'origin/main' into device-editor

7bd3f51

knolleary mentioned this pull request May 3, 2023

Tests failing following semver 7.5.0 update #2077

Closed

Steve-Mcl added 4 commits May 4, 2023 13:39

defaultValue in the DB upgrade should match spec

499214d

ensure features are guarded by permissions

3d98b81

and for licensed platform only

correct misleading api route info

530f275

initial round of tests

b184598

Steve-Mcl marked this pull request as ready for review May 4, 2023 13:33

Steve-Mcl requested a review from knolleary May 5, 2023 12:55

knolleary added the area:migration Involves a database migration label May 9, 2023

Steve-Mcl and others added 3 commits May 9, 2023 16:43

add note in docs about needing to have a snapshot

2437ea4

mqtt config note about remote Device Editing

c901ea1

Refactor device editor endpoints and move under ee code tree

aef11ef

knolleary mentioned this pull request May 9, 2023

Refactor device editor endpoints and move under ee code tree #2109

Merged

11 tasks

Steve-Mcl and others added 2 commits May 9, 2023 23:17

Merge pull request #2109 from flowforge/1821-refactor-device-editor-api

bb77a15

Refactor device editor endpoints and move under ee code tree

fix create snapshot button text wrapping

f82ad24

knolleary reviewed May 10, 2023

View reviewed changes

frontend/src/pages/device/Overview.vue Outdated Show resolved Hide resolved

Update frontend/src/pages/device/Overview.vue

b3e9d3c

Co-authored-by: Nick O'Leary <nick.oleary@gmail.com>

Steve-Mcl mentioned this pull request May 10, 2023

Audit log entries for developer mode and remote editor access #2115

Closed

Update device docs

726ca5b

knolleary approved these changes May 10, 2023

View reviewed changes

Steve-Mcl commented May 10, 2023

View reviewed changes

docs/user/devices.md Show resolved Hide resolved

Steve-Mcl commented May 10, 2023

View reviewed changes

knolleary added 2 commits May 10, 2023 10:42

Fix docs link

18de4eb

Add better checking for unsupported dev agent version

a7252ef

knolleary approved these changes May 10, 2023

View reviewed changes

knolleary merged commit b2f882b into main May 10, 2023
3 of 4 checks passed

knolleary deleted the device-editor branch May 10, 2023 09:43

knolleary mentioned this pull request May 10, 2023

FlowForge should show device agent version in the dashboard #2047

Closed

knolleary linked an issue May 10, 2023 that may be closed by this pull request

FlowForge should show device agent version in the dashboard #2047

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remote device access, editing and snapshotting with Auth enabled #2042

Remote device access, editing and snapshotting with Auth enabled #2042

Steve-Mcl commented Apr 25, 2023 •

edited

Loading

Steve-Mcl commented May 2, 2023

knolleary commented May 2, 2023

Steve-Mcl commented May 4, 2023

knolleary commented May 9, 2023

Steve-Mcl commented May 10, 2023

Steve-Mcl May 10, 2023 •

edited

Loading

	async function assignDeviceToProject (device, project) {
	// #region Helpers
	async function assignDeviceToProject (device, project) {

Remote device access, editing and snapshotting with Auth enabled #2042

Remote device access, editing and snapshotting with Auth enabled #2042

Conversation

Steve-Mcl commented Apr 25, 2023 • edited Loading

Description

TODO:

Notes for debugging and comprehension

Related PRs

Related Issue(s)

Checklist

Labels

Steve-Mcl commented May 2, 2023

Security

Enterprise Only

UI / UX

knolleary commented May 2, 2023

Steve-Mcl commented May 4, 2023

knolleary commented May 9, 2023

Steve-Mcl commented May 10, 2023

Steve-Mcl May 10, 2023 • edited Loading

Choose a reason for hiding this comment

Steve-Mcl commented Apr 25, 2023 •

edited

Loading

Steve-Mcl May 10, 2023 •

edited

Loading