AI Function Node Iteration

[joepavitt]

Description

• Build a REST API service that wraps the same intelligence running in the function-gpt node.
• Build a new function-ai?? node that will run on all NR instances on FlowFuse Cloud (and maybe self-hosted if we can work out smooth way of deploying)
• Plugin for Node-RED that tightly integrates with the editor
  • https://github.com/FlowFuse/admin/issues/320
  • https://github.com/FlowFuse/nr-assistant
• Support instances
  • Add runtime settings to nr-launcher for nr-assistant nr-launcher#255
  • Add API to FlowFuse core for handling assistant requests #4083
• Automation, integration, configuration
  • Add the templated settings to the flowforge.yml file for nr-assistant configuration helm#425
  • Automations for nr-launcher releases nr-assistant#6
  • update .github/workflows/publish.yml for nr-assistant nr-launcher#259
  • FlowFuse/CloudProject#457
    • Staging
    • Production
• Support devices
  • Add necessary config settings to device-agent for nr-assistant
  • Update FF snapshot to include nr-assistant package in snapshot
• Follow up tasks
  • User documentation for "The FlowFuse Assistant" #4101
  • Changelog entry for "The FlowFuse Assistant" website#2286

Which customers would this be available to

All

Have you provided an initial effort estimate for this issue?

I have provided an initial effort estimate

[Steve-Mcl]

Design

Having been working on the 1st part (API) and I have stuck with Open AI.

Using functions (example: https://www.datacamp.com/tutorial/open-ai-function-calling-tutorial) I have got it to return parsable data including any additional libs

example response:

This lead me to the thought - what if we didnt have a 3rd full duplicate of the core function node but instead, made this available to FF based Node-RED instances via a plug in?

The plugin, when called could return all of the elements required to simply generate a new function node for the canvas.

Additionally, when the function node is already created in the workspace, we can still call the API via the plugin by adding a prompt in the monaco editor.

Thoughts?

[joepavitt]

So, from a UX perspective, two entry points:

• Plugin sidebar? Ask any request, get a new function node
• Core function node has extra method/input in it as per our node and they can interface with that?

It feels more intuitive and obvious to a user if it's an explicit text input available to use, rather than them having to write comment-based prompts for example

[joepavitt]

Agree with the plugin direction, rather than another node. Coupling tightly with existing function node would be preferable

[ZJvandeWeg]

@joepavitt Which tier of customer is this available to?

[joepavitt]

Would be all - not sure why that was missed

[Steve-Mcl]

Just sharing some progress and a request for feedback

Current operational state (rough n ready)

Generating a function node

Generating a flow

Generating JavaScript while inside a function

Notes

You might wonder why there is a separate "Flow Builder" and "Function Node Builder"...
In short, asking the AI to make a function node is super reliable. I provide system prompts to generate plain JavaScript (suitable for a node-red function) and use the OpenAI function + schema to respond with a JS object containing number of outputs, function code, any npm modules it may have used etc. Then, in FF I hydrate that into a well formed Node-RED JSON array containing a single function node. This works very well.

The flow builder is more more suseptible to "halucinations" and gets thigs wrong very often

Lastly the editor "Ask FlowFuse 🪄" uses something called a Code Lens (monaco feature) and calls to the same endpoint as the Function Node builder. This means we can do similar things like adjust the output count and add libs to the setup tab.

Accessing these menu items (please vote which option)

I could not decide (discussed with @joepavitt) how to surface these. we agreed i would post screenshots for comparison.
Please take a look and chose what feels best.

Option 1 - front and center (not easy to miss)

Option 2 - top level side menu

Option 3 - added to an existing side sub menu (I chose "view", but it could be any)

Other feedback would be great too

The menu items, dialogs titles / descriptions of the feature are very much thown together. Some feedback and suggestions are welcome.

• Global operations (flow builder / function Node builder)
  • Menu item titles
  • Dialog titles
  • Prompt descriptions on dialog body
  • AI Magic Icon in top header (linked issue Node-RED Icon: For AI operations website#2258)
• Inline code action
  • "Ask flowfuse 🪄`" Good? Bad? Other?
  • Dialog title
  • Prompt description on dialog body

[Steve-Mcl]

Future iteration though dump

Via Monaco Editor...

1. Code lens could be used on JSON editor - "Generate an array of 10 objects containing random names and dates"
2. Code lens could be used on the template editor for the various code styles (JSON, CSV, JavaScript, SQL)
3. Code Lens could be used in the Dashboard 2.0 ui-template for assistance building templates (untested at this time)

Additional menu items:

1. Select a bunch of nodes and "Ask FlowFuse what this flow does"
2. More specific prompts that can have fine tuned system hints like "Add an endpoint", "Make a HTTP Request"
   3. Under the hood, this would work similar to the function node builder where specific system hints asked for specific structured response then we hydrate the response into a well formed flow

[Steve-Mcl]

Naming things...

Taking suggestions for module/repository name for this plugin?:
@flowfulse/nr-ai-plugin (similar to nr-tools-plugin)
@flowfulse/nr-ai-assistant
@flowfulse/nr-ai-assist
@flowfulse/node-red-ai-plugin
other?

[knolleary]

Given the hallucinations, lets keep the scope on the Function node for this iteration (so we can get it released this week) and do the flow builder for v2.

Of the three options, I think Option 1 is probably the best to begin with to get this feature noticed. Burying it under the menu will make it hard to discover.

The Code Lens piece looks really nice 👍🏻

For naming, I'd suggest @flowfuse/nr-assistant - I could see this expanding out to other features which don't necessarily have to be anything to do with AI, but are there to help the user.

Where can I see details of the rest api this uses? We need to make sure the security around it is appropriate.

I presume this will work on the Device Agent as well?

[Steve-Mcl]

@knolleary I had written most of the below when your comment landed (so it may seem a bit mixed up.

Where can I see details of the rest api this uses? We need to make sure the security around it is appropriate.

Regarding integration with FF platform.

currently, this plugin is under local development and uses an endpoint hosted in a Node-RED instance on FFC. This make iteration and improvement super quick.

However, to implement this securely, currently, the endpoint is protected using a single token setup on the instance security settings.

I cannot immediately think of a way to avoid having to bring this endpoint and API into the FF platform.

Can I grab you for 5 mins to discuss a scalable and suitable direction for integration with FF core please?

I presume this will work on the Device Agent as well?

It was not mentioned in scope and is a question I think you have just answered ;)
I also think that ties up with the previous point above.

For naming, I'd suggest @flowfuse/nr-assistant

Yes, makes sense & I love it too (bonus).

Given the hallucinations, lets keep the scope on the Function node for this iteration (so we can get it released this week) and do the flow builder for v2.

I agree completely. will remove that menu item for 1st iteration.

Of the three options, I think Option 1 is probably the best to begin with to get this feature noticed. Burying it under the menu will make it hard to discover.

Perfect.

[knolleary]

Summary of a discussion @Steve-Mcl and I had:

For first iteration:

1. Adding api to core platform to expose the assistant apis. Will be secured based on instance/device token.
2. OpenAI config provided in platform yml file to enable this feature

This will then allow us to enable this feature for FFC users.

For self-hosted, in the first iteration, they will be able to provide their own OpenAI tokens in their yml file to unlock this feature.

Future iteration - provide a public API that self-hosted instances can access to enable this feature. The api request will include their license identifier which we can use to validate the request is a valid customer.

Note - self-hosted on air-gapped networks will not be able to make use of this feature.

[Steve-Mcl]

Regarding: HTTP requests made by the nr-assistant plugin (from runtime -> AI API)

In PoC, I simply used Node built-in fetch which works fine however we have some users stuck on node 14/16 (on devices?) and fetch is Node 18 min (and even then it is experimental until Node 21)

I noted that nr-tools uses the nifty little undici lib - but this is around 1.2mb installed and currently requires node >= 18.17

There is also a super small (160kb installed) ky (by the got folk) but it would be yet another/different lib

So I have decided to stick with got for the plugin for the first iteration but I am struggling to justify NOT going with v11 of got

Do we:

• Stay with v11 + require (like device agent, nr-files, nr-project-nodes and some others)
• OR
• Follow Node-RED lead and go with v12.6.0 + dynamic import
• OR
• undici (like nr-tools) meaning users of the plugin must be on Node 18.17 min

My gut says since the plugin runs in the Node-RED runtime we should match up with Node-RED & use 12.6.0 + dynamic import HOWEVER that means the plugin is limited to later versions of node-red making me think we should take the technical debt hit and go with got 11 like project nodes and file nodes.

[knolleary]

Just to be accurate, nr-tools uses undici 5.x which supports older Node.js versions.

got v11 is tried and tested. It isn't necessary my first choice these days, but lets stick with what works.

[knolleary]

@Steve-Mcl we will need an issue on the helm repo to add the templated settings to the flowforge.yml file - this is what the cloudproject issue will then be able to make use of when we publish the release.

[Steve-Mcl]

@knolleary raised here: FlowFuse/helm#425

[joepavitt]

@Steve-Mcl can you drop you status update here when available please?

[Steve-Mcl]

@Steve-Mcl can you drop you status update here when available please?

• API in FF merged
• basic automations have just been merged in nr-assistant.
  • That means i can now tag a release (immediate next task)
  • That unblocks the nr-launcher PR Map runtime settings for nr-assistant nr-launcher#256 (Next task following above)
• follow up tasks for release process are being kindly handled by @ppawlowski
  • Automations for nr-launcher releases nr-assistant#6 (discussed)
  • Add the templated settings to the flowforge.yml file for nr-assistant configuration helm#425 (in progress)
  • https://github.com/FlowFuse/CloudProject/issues/457 (thursday item)

following that, I will be finishing up device agent integration - but that is non blocking of the release for instances.

[Steve-Mcl]

Verified on Production

[joepavitt]

Closing this out as it's now in production

[Steve-Mcl]

Closing this out as it's now in production

Should we reopen for device agent support or generate new issue Joe?

[knolleary]

@Steve-Mcl new issue that is specifically scoped to agent support