Preinstall Palette Nodes

[sammachin]

Epic

#141

Description

As a: user

I want to: specify node that are pre-installed as part of a template

So that: I can start a project with my commonly used nodes

Acceptance Criteria

• Template has list of 'pre-install modules'
• strings (npm package names) can be added or removed from the list
• version can be defined with @1.2.3 syntax
• ...

[sammachin]

Considerations:
Is this limited to just node-red packages or is there a valid use case for any npm module? do we care?
Should there be any validation that its a valid package name and version when the template is created? maybe one to iterate?

[knolleary]

Adding a list of npm module specifiers (name/version) in the Template object is fairly straight forward.

The harder parts are:

Getting the UI right

Although a simple expandable list of text inputs (like the env var table) should do for MVP.

Considering how the nodes then get installed

Given the memory constraints a container could be running under, we know that running npm install can cause memory usage spikes that exceed what the flows would otherwise be doing. In other words - it takes more memory to install something than it does to run it. That can lead to usability issues with crashing projects.

It is always more preferable for the nodes to get 'baked' into the container image being run - however that is an administrative task to build such an image, then create a stack that identifies the new image and make it available. That is hardly the self-service model we want in ff.cloud.

When nr-launcher is asked to start NR and it retrieves the project settings that now includes a list of nodes, it should compare that list against its local package.json to see if anything new is required - and if so, run the suitable npm commands to install them. (Should also remove non-default modules not in that list) Only once that is done can it start Node-RED.

That will cause a slower startup time - need to ensure we don't get timeouts in the forge UI/API whilst waiting for it to complete.

[sammachin]

@knolleary I assume the slower start time is the tradeoff that frees up more memory for NPM because Node-RED isn't running?

[knolleary]

It is generally better to have the node installed before Node-RED starts. In fact, the 'auto install' capability only works for nodes that Node-RED knows it needs to have installed based on what it saw the last time it was running. It wouldn't work for the case a user chooses to add a node via the FF dashboard - unless we also manually update the NR runtime settings in the database to make it think it knows about it. (Sorry if that isn't clear... I know what I mean and these comments will make sense to me when we look at this properly).

Another consideration that may be inscope of this story is:

if the FF project page lists the preinstalled nodes, should it also list nodes the user has installed for themselves via the Manage Palette option in the editor?

I think there should be a single consolidated view - we wouldn't want things to get confused because there are two different ways to add nodes to the project and they result in different views saying different things.

[sammachin]

ignore me!

@knolleary does any of this change now that we have stacks? I think these discussions pre-dated stacks, and am I right in thinking that pre-installing nodes would be a stack thing not a template thing?

[hardillb]

This has some tie in with FlowFuse/nr-launcher#54 and FlowFuse/nr-launcher#53 (comment)

[knolleary]

Revisiting this story, I'm increasing the sizing as there as a reasonable amount of UI work to get right as well as launcher work to get the nodes installed. Plus there are, of course, some edge cases that need thinking about.

• In the FlowForge UI, it is possible to specify a list of node modules (+versions) that the project should have available.
• A Template can specify a list of nodes
• A Project inherits that Template and allows the user to change the list - either modifying existing entries, or adding additional ones.
• When the launcher starts, it gets the consolidated list of nodes, ensures package.json reflects the right information and then runs npm install - and only then starts Node-RED.

The problems start when you consider the Palette Manager in the editor - where a user can both install and remove nodes within the editor directly.

Essentially: How do we resolve changes made in the flowforge UI with changes made in Node-RED directly?

This can get quite complicated. To keep it simple, I think the approach I favour is:

• The Template might provide a boot-strap list of nodes
• Assuming the Template has Allow user to install new nodes set to true, the Project can modify the list of nodes - but it will require a restart of Node-RED for any changes to get applied
• If the user uses the Palette Manager, I think can see a way to trigger an update of the Project Settings to reflect any changes made. That will keep the views in sync.

The only gap is what happens if the user has modified the node list in the project, not restarted NR, then uses the NR palette manager to modify the list. We now have two sets of competing changes... I suspect we just need to start implementing all of the above and the right approach to this bit will emerge.

[hardillb]

Verified on Staging

[Steve-Mcl]

Slight issue with the input overflowing...

Chrome is up to date
Version 106.0.5249.119 (Official Build) (64-bit)

[JMDaling]

Hi @knolleary, on your second point in your comment above you mention specifying a template. Where exactly do you do this?

My goal:
To migrate existing node-red palette configurations (by copying the package.json dependency list and not using the FF-GUI)

What I tried:

• Stopped the node-red instance on FF
• Identified the project and copied the dependencies from another installation into its package.json file
• Start up the instance on FF

This resulted in an overwrite of the package file.

After adding the dependencies and doing an npm install manually, it installs the packages as expected but after a restart of the FF instance again it is reverted in the settings of the project.

I also attempted to activate the autoInstall configuration in the settings.js file. It also gets overwritten.

Comments on this will be appreciated.

[knolleary]

Hi @JMDaling

I assume this within your own local install of FlowForge? We do not support manually editing the individual instance's package.json and settings files - they are all managed/generated by the platform.

Currently, the only ways to manage the packages in your instance is either via the Manage Palette option within the editor, or via the Instance palette settings view in the platform.

We do have an item in the backlog to support adding packages to the Instance Template - so they get preinstalled with any instance created from that Template. You can follow #2002 for updates.