Provide persistent context options

[knolleary]

Description

The default Context storage in Node-RED is memory only.

A value-added feature is to support persistent context storage options - such as via Redis.

A possible extension would be to support sharing the context store between two projects in the same team - useful for high-availability type scenarios.

Which customers would this be availble to

• All Users, (CE)
• Specific Project Types (EE)
• Specific Team Tiers (EE)
• Other (See comments)

[sammachin]

Initial user feedback is showing strong support for this feature

[sammachin]

Another request from a user for this mentioned as a blocker for migrating Node-RED projects to flowforge

[sammachin]

What should happen to persistent context data when a project is suspended, should we clear the data or not?

[knolleary]

There are three key design points for this item:

1. what backend storage mechanism is used
2. how this storage layer handles authentication
3. how this applies to Devices-vs-Projects

We added the File Server component to the platform in 1.1.0. This already solves the authentication question and I propose we extended it to also provide a Context API. That solves item 2.

For item 3, Devices should be configured with a standard built-in file based context - they do not need to access the File Server.

That leaves Item 1 - what backend storage we use for context requests. When running locally, it can still be a local file based implementation. When running in the cloud, I don't think it make sense to leverage the S3 api - S3 isn't really suitable for the sort of traffic context could generate and would get expensive. Redis has long been the assumed best service for Context - what options are available for the k8s and docker deployments?

[Steve-Mcl]

When running in the cloud, I don't think it make sense to leverage the S3 api - S3 isn't really suitable for the sort of traffic context could generate and would get expensive

What if (similar to the NR file backed context) we use a memory cache & flush every 30? Is that still too frequent?

[knolleary]

What if (similar to the NR file backed context) we use a memory cache & flush every 30? Is that still too frequent?

It depends on what quality of service we want to provide. Having a potential 30s data loss window could be a problem.

But equally, for an initial iteration, it could be okay if it removes the need to introduce another data service. Before we commit to a path however, we should understand the $ costs of S3 - it isn't really intended for this type of usage so its pricing could well penalise us quite heavily.

We can make a start on all the other bits whilst we validate the choice of service.

[Steve-Mcl]

(thinking out loud / definitely lateral / one for discussion?...)

I wonder if we could utilise persistent MQTT topics in the (existing) broker as the cache & flush less frequently?
Any project crash would then not result in cache loss. (though broker crash would lose cache). 🤔

NB: Will be speaking with Ben very shortly (slack huddle) about this epic.

[hardillb]

MQTT brokers should not be used as ad-hoc object stores

Current consideration is to build something very similar to the file-server, but using Redis (or similar) as the backend

[hardillb]

Will look at the 2 different AWS services that provide Redis support https://aws.amazon.com/redis/

• MemoryDB
• ElasticCache

Also need to have another look for Postgres HStore client for NodeJS, all the ones I've found so far are abandonded.

After talking to Nick, I will add some more endpoints to the file-server rather than start another new project

[Steve-Mcl]

EDITED 12/08/2020

See latest thread

[hardillb]

First pass at context api

https://github.com/flowforge/flowforge-file-server/tree/persistence-context

[ZJvandeWeg]

project settings (enable/disable)

@Steve-Mcl I'd rather not have more options. Either this improves something and it's the only way forward, or we don't build the feature at all.

[Steve-Mcl]

project settings (enable/disable)

@Steve-Mcl I'd rather not have more options. Either this improves something and it's the only way forward, or we don't build the feature at all.

ZJ, This is referring to how we enable / disable the context plugin automatically due to env (e.g. cloud vs device) - not a user option.

The above points were only ever intended to be be reminders for myself and @hardillb to come back and pad these out with details once the initial "first pass" was done. Will try to be more explicit in future.

[Steve-Mcl]

Status as of 29/11/2022

(actions complete, expand to see)

• Ben: first pass at API (in file-server) with basic mem backend
• Steve: Other priority tasks (NR 3.1 related) (28th / 29th morning)
• Ben: Push to private repo (29th Nov afternoon)
• Steve: Pull & 1st implementation of NR context plugin to utilise (29th Noon / 30th Morning)

Status as of 5/12/2022

(actions complete, expand to see)

• Ben to implement clean API at backend
• Default name for this store to be persistent
  • Additionally, the memory store should be named memory and be set as default
• ship minimal version on staging by Wednesday 7th
  • Document any limitations or temporary workarounds along with 1st push (awaiting final status)
    • deferred to new action list below
  • Follow up meeting to discuss the feedback and direction DONE 8/12/2022
• PR device agent to have 2 default context stores persistent & memory (to match cloud project)
  • This is prepared by adding "memory" and "persistent" in flowforge-device-agent/lib/template/template-settings.js however there are 2 other ways to implement this. Need direction.
    • (x) in flowforge-device-agent/lib/template/template-settings.js Decided 8/12/2022
    • ( ) in ./settings.json
    • ( ) in forge application (added to the settings.js file delivered to the device)
    • PR Raised 8/12/2022 Add "memory" and "persistent" as context stores device-agent#44
• Following 1st release practical tests on staging, address the issue of whether to implement a cache to support sync functions: Discussed 8/12/2022. Moved down to Outstanding Actions

Status as of 8/12/2022

(current)

• Local testing of Redis and Postgres context drivers complete
• Testing on staging complete (memory backed in file-server only)
• Arrange meeting as discussed 5/12/2022 to cover remaining items (DONE - new actions below)

Outstanding actions (post 8/12/2022 meeting: @Steve-Mcl, @hardillb & @knolleary in attendance)

• Choose default backend (Ben is determining cost + effort vs benefit. TBD 9/12/2022)
  • Postgres is the chosen backend
• Tests to be written
  • node-red plugin (Steve 8th/9th)
    • Add tests nr-persistent-context#8
    • merge PR
  • backend
    • Context API with memory backend (Add context api tests file-server#28)
      • merge PR
    • Context API with sequelize backend (add tests for database backed context file-server#32)
      • merge PR
    • Context API with redis backend
      • merge PR
    • File API with memory backend (fix tests and make sure they run for PRs nr-file-nodes#15)
      • merge PR
    • File API with localfs backend (fix tests and make sure they run for PRs nr-file-nodes#15)
      • merge PR
    • File API with s3 backend
      • Follow up issue added Add tests for s3 backend storage file-server#40
    • Ensure quota cannot be exceeded (Calculate quota in driver to ensure correct size file-server#35)
      • merge PR
• Documentation
  • user facing (what it is, what its limitations are (async in Funcs) (Steve) (added to readme)
  • Add section "Working with files and persistent context" to https://flowforge.com/docs/user/ (Document persistent context and file storage #1465)
  • Update feature list (Update File, Context & SSO on the feature comparison website#294)
  • install / upgrade / setting quotas docs (Ben) (Cover configuring context and setting quotas #1474)
  • Readme for plugin
  • Readme for file-server
• Q: Do we add cache layer for sync operations?
  • A: Needs sizing to understand if it can be done before 1.2 release
  • Raise story for next iteration (Cannot use sync operations for context storage nr-persistent-context#10)
• Quota control?
  • Minimum: stories to raise for both context and file storage
    • Raise story for file quota (Steve)
    • Raise story for context quota (Ben)
  • Better handling of failures HTTP 413 errors are sent for over quota (Steve)
    • context plugin (improve errors for node-red, incl 413 payload size nr-persistent-context#9)
    • file-nodes (improve errors for node-red, incl 413 payload size nr-file-nodes#14)
  • Defaults to be set (Decided 19/12/2022) (Add context store provisioning helm#79)
    • 1MB context storage quota limit
    • 100MB file storage quota limit
• Helm charts to update (Ben)
• Merge device-agent update for adding "memory" and "persistent" context stores: Add "memory" and "persistent" as context stores device-agent#44

Items to clarify and plan out.

• project settings (enable/disable) Decided 5/12/2022: presence of EE licence determines enable/disable (DONE)
• naming Decided 5/12/2022: "memory" and "persistent" to be added to cloud and device settings (DONE)
  • device agent should have matching store which is original (actual) file store PR DONE
• design decision on when to delete that store (potentially when project ID changes)
• will it be default context? No
• quota considerations Discussed 8/12/2022: See Outstanding Actions 8/12/2022
  • mem + storage
• improve logging for capturing use/overuse
• TBD production backend(s) TBD over coming days. (evaluation & discovery ongoing) - Discussed 8/12/2022
  • Currently, (for testing) a memory and Redis backend have been implemented - Decision to be made 9/12/2022
  • Use of the existing FF database to be considered before decision is made - Decision to be made 9/12/2022

[knolleary]

As this issue is in the Done column on the dev board and was delivered in 1.2 can it be closed? Any follow-on items should be raised as separate issues at this point.

[Steve-Mcl]

Yes, I believe now is ok - we have raised issues for the timeout problem and s3 tests (which are now in a PR & I am currently testing the minio stuff locally)