An experiment to replace Node-REDs existing NodeJS backend with an Erlang equivalent that is 100% compatible to existing flow code.
The goal is bring the advantages of low-code visual flow-based programming to a programming language that is designed for message passing and concurrency from the ground up, hence Erlang. More details described in the corresponding blog post.
Breadboards are prototyping devices found in electronics. Erlang-Red can be best thought of as a programming breadboard.
What are some tools for software prototyping? Besides AI and VScode. Software developers create prototypes but they don't prototype software.
A telnet session flow describes how breadboard programming can be done using Erlang-Red. That flow prototypes a possible software solution starting with a simple concurrent approach until a first final approach is found. All solutions are testable and usable - instantly and all solutions build on previous solutions - simply copy and paste the flows. That's prototyping.
Node-RED is great for creating data flows that actually describe concurrent processing, it is just a shame the NodeJS is single threaded. So why not use something that is multi-process from the ground up? Concurrency is guaranteed and included.
Also Erlang isn't the most understandable of programming language - unless one has fallen into in a cauldron of Prolog, spiced with Lisp.
So won't it be great to have the simplicity of low-code visual flow based programming and the performance (and concurrency) of Erlang?
My development process is best described as flow driven development based around a set of test flows to ensure that node functionality is implemented correctly - meaning that it matches the existing Node-RED functionality.
Test flows are mirrored in a separate repository for better maintainability and also integration with existing Node-RED installations.
Erlang architecture is best described by describing various use cases:
- Deploying flows to Erlang-Red. Explains the start up process and how Erlang processes are started for nodes.
- Workings of a supervisor node supervising a function node.
- The challenges of function node which must support timeouts, sub-processes and being supervised by a supervisor.
- Inner workings of link nodes and how to deal with dynamic link calls.
This is a non-complete list of nodes that partially or completely work:
| Node | Comment | Example Flow |
|---|---|---|
| catch | catches exception of selected nodes and of entire flows but not groups | Flow |
| change | supports many operators but not all. JSONata in basic form is also supported. | Flow |
| complete | is available and can be used on certain nodes, not all | Flow |
| csv | initial RFC4180 decoder working, supports only comma separator | Flow |
| debug | only debugs the entire message, individal msg properties aren't supported. msg count as status is supported. | Flow |
| delay | supported static delay not dynamic delay set via msg.delay |
Flow |
| exec | executing and killing commands is supported but only for commands in spawn mode and set on the node. Appending arguments to commands isn't supported. Timeouts are supported. Kill messages are also supported. | Flow |
| file in | working for files located in /priv |
Flow |
| function | working for any Erlang. Stop and start also respected. Timeout and more than one output port isn't supported. | Flow |
| http in | working for GET and POST, not available for PUT,DELETE etc | Flow |
| http request | basic support for doing rrequests, anything complex probably won't work | Flow |
| http response | working | Flow |
| inject | working for most types except for flow, global ... | Flow |
| join | manual arrays of count X is working, parts isn't supported |
Flow |
| json | working | Flow |
| junction | working | Flow |
| link call | working - dynamic & static calls and timeout is respected | Flow |
| link in | working | Flow |
| link out | working | Flow |
| markdown | working and supports whatever earmark supports. | Flow |
| mqtt in | should be working | Flow |
| mqtt out | should be working | Flow |
| noop | doing nothing is very much supported | Flow |
| split | splitting arrays into individual messages is supported, string, buffers and objects aren't. | Flow |
| status | working | Flow |
| switch | most operators work along with basic JSONata expressions | Flow |
| tcp in | Tcp in node supports starting a TCP/IP server listening on a specific port. | Flow |
| tcp out | Tcp out node that currently only supports the reply-to node to respond to an existing tcp in connections. | Flow |
| template | mustache templating is working but parsing into JSON or YAML isn't supported | Flow |
| trigger | the default settings should work | Flow |
These nodes represent specific Erlang features as nodes and as such, could be implemented in NodeJS to provide Node-RED with the same functionality.
| Node | Comment | Example Flow |
|---|---|---|
| event handler | Erlang-Red node for the Erlang gen_event behaviour. Supports both dynamic and static configuration of the event handler. |
Flow |
| module | Erlang module for defining Erlang modules that can be used with the function, event handler and statemachine nodes. | Flow |
| supervisor | Erlang-only node that implements the supervisor behaviour. Supports supervising supervisors and ordering of processes (i.e. nodes) to ensure correct restart and shutdown sequences. | Flow |
| statemachine | Implements the gen_statem behaviour. Requires a module node to define the actions of the statemachine. |
Flow |
These nodes can be installed using the corresponding Node-RED node package.
Nodes for ensuring truth in unit test flows.
| Node | Comment | Example Flow |
|---|---|---|
| assert failure | Sending this node a message, will cause test failure. This node ensures certain pathways of a flow aren't reached by messages. | Flow |
| assert success | If this node isn't reached during a test run, then that test will failure. This node represents pathways that must be traversed. | Flow |
| assert debug | This node can be used to ensure that another node produces content for the debug panel. | Flow |
| assert status | Ensure that a node is assigned a specific status value. | Flow |
| assert values | Check specific values on the message object and ensure these are correct. | Flow |
These nodes can be installed using the corresponding Node-RED node package.
- Contexts are not supported, so there is no setting things on
flow,nodeorglobal. - JSONata has been partially implemented by the Erlang JSONata Parser.
Elixir helpers can be added to erlang-red-elixir-helpers repository.
There is nothing stopping anyone from creating a complete node in Elixir provided there is a Erlang "node-wrapper", i.e., a bit of Erlang code in the src/nodes directory that references the Elixir node.
The initial example markdown node is an Erlang node that references Elixir code. I also wrote an Elixir wrapper function whereby I could have just as easily referenced Earmark directly from the Erlang code. That was a stylist choice.
I intend to use Elixir code for importing Elixir libraries to the project and less coding nodes in Elixir. I simply prefer Erlang syntax. But each to their own :)
$ rebar3 get-deps && rebar3 compile
$ rebar3 eunit
$ rebar3 shell --apps erlang_red
Open the Node-RED visual flow editor in a browser:
$ open -a Firefox http://localhost:9090/node-red
I use docker to develop this so for me, the following works:
prompt$ git clone git@github.com:gorenje/erlang-red.git
prompt$ cd erlang-red
prompt$ docker run -it -v $(pwd)/erlang-red:/code -p 9090:8080 -w /code --rm erlang bash
docker> rebar3 shell --apps erlang_red
Then from the docker host machine, open a browser:
prompt$ open -a Firefox http://localhost:9090/node-red
That should display the Node-RED visual editor.
A release can be bundled together:
$ rebar3 as prod release -n erlang_red
All static frontend code (for the Node-RED flow editor) and the test flow files in priv/testflows are bundled into the release.
Cowboy server will started on port 8080 unless the PORT env variable is set.
A sample Dockerfile Dockerfile.fly is provided to allow for easy launching of an instance as a fly application.
The provided shell script (fly_er.sh) sets some common expected parameters for the launch.
Advanced users may wish to examine the fly launch line therein and adjust for their requirements.
Using the container stack at heroku, deployment becomes a git push heroku after the usual heroku setup:
heroku login-->heroku git:remote -a <app name>-->heroku stack:set container-->git push heroku
However the Dockerfile.heroku does not start the flow editor, the image is designed to run a set of flows, in this case (at time of writing) a simple website with a single page.
Basically this flow is the red-erik.org site.
The image does this by setting the following ENV variables:
COMPUTEFLOW=499288ab4007ac6a- flow to be used. This can also be a comma separated list of flows that are all started.DISABLE_FLOWEDITOR=YES- any value will do, if set the flow editor is disabled.
Also be aware that Erlang-Red supports a PORT env variable to specifying the port upon which Cowboy will listen on for connections. The default is 8080.
Heroku uses this to specify the port to connect for a docker image so that its load balancer can get it right.
What the gif shows is executing a simple flow using Erlang as a backend. The flow demonstrates the difference in the switch node of 'check all' or 'stop at first match'.
All nodes are are processes- that is shown on the left in the terminal window.
This example is extremely trivial but it does lay the groundwork for expansion.
To create unit tests for this, Node-RED frontend has been extended with a "Create Test Case" button on the export dialog:
Test flows are stored in the testflows directory and will be picked up the next time make eunit-test is called. In this way it is possible to create unit tests visually.
Flow tests can also be tested within the flow editor, for more details see below.
The flow test suite is now maintained in a separate repository but is duplicated here.
To better support testing of flows, two new nodes have been created:
"Assert Failed" node cases unit tests to fail if a message reaches it, regardless of any message values. It's basically the same as a assert(false) call. The intention is to ensure that specific parts of a flow aren't reached.
The second node (in green) is an equivalent to a change node except it contains test on attributes of the message object. Possible tests include 'equal', 'match', 'unset' and the respective inverses. Here the intention is that a message passes through is tested for specific values else the unit test fails.
These nodes are necessary since there is no other way to test whether flow is working or not.
Also remember these flow tests are designed to ensure the Erlang backend is correctly implementing node functionality. The purpose of these nodes is not to ensure that a flow is correct, rather that the functionality of implemented nodes works and continues to work correctly.
My plan is to create test flows that represent specific NodeRED functionality that needs to be implemented by Erlang-Red. This provides regression testing and todos for the implementation.
I created a keyboard shortcut for creating and storing these test flows from the flow editor directly. However I was still use the terminal to execute tests make eunit-test - which became painful. So instead I pulled this testing into Node-RED, as the gif demonstrates:
What the gif shows is my list of unit tests, which at the press of a button, can all be tested. Notifications for each test shows the result. In addition, the tree list shows which tests failed/succeed (red 'x' or green check). Also tests can be executed individually so that failures can be checked individually.
The best bit though is that all errors are pushed to the debug panel and from there I get directly to the node causing the error. Visual unit testing is completely integrated into Erlang-Red.
My intention is to create many small flows that represent functionality that needs to be implemented by Erlang-Red. These unit tests shows the compatibility to Node-RED and less the correctness of the Erlang code.
Contributions very much welcome in the form of Erlang code or as Node-RED test-flows, ideally with the Erlang implementation. Elixir code is also welcome, only it has its own home.
Each test flow should test exactly one feature and use the assert nodes to check correctness of expected results. Tests can also be pending to indicate that the corresponding Erlang functionality is still missing.
An overview of the sibling projects for both the reader and me:
- Unit test flow suite provides visual unit tests that verify the functionality being implemented here is the same as in Node-RED. Those test flows are designed to be executed in both Node-RED and Erlang-Red. FlowHub.org maintains the repository and is used to synchronise flow tests between Erlang-Red and Node-RED. These tests can also be used for other projects that aim to replicate Node-RED functionality in an alternative programming language.
- Node-RED and Erlang-Red unit testing nodes are used to define and automatically ensure the correct functionality. These nodes are embedded in test flows and ensure that test flows are correct. This makes testing repeatable and reliable and fast! As an aside, these nodes are maintained in an Node-RED flow.
- JSONata support for Erlang-Red is implemented by an Erlang parser with a grammer that covers most of JSONata syntax, no guarantees made. Support of JSONata functionality is limited to what the test flows require. Nothing prevents others from extending the functionality themselves, it is not a priority of mine.
- Elixir helper library allows Elixir code to be also part of Erlang-Red. Erlang-Red is not intended to be a pure Erlang project, it is intended to be a pure BEAM project. Anything that compiles down to the BEAM VM, why not include it?
- Supervisor nodes and other Erlang behaviours as Node-RED nodes. Node package includes
gen_statemandgen_eventas nodes that can be used with Erlang-Red flows. These nodes can also be installed into Node-RED but there they do nothing.
Questions and Answers at either the Erlang Forum or the Node-RED Forum.
Also for more details, there was also a discussion on Hacker News.
To branch or not to branch, that isn't really a question. I'm currently working directly on main but ensure that all tests succeed before pushing, so main branch will always work. Locally I work with branches but have no desire to make those branches public since I'm working on my own.
Versioning is completely random and has little or no meaning at the moment. I prefer to use Milestones since these are arrived at and are not planned, I don't know when the next milestone will be reached.
If this project becomes more collaborative or a "production ready piece of software", more certainty will be applied to the development process, i.e., semantic version numbers will introduced.
I'm more than happy to deal with conflicts if someone developed something on a branch and it doesn't merge - I understand that multiple direct pushes to main everyday isn't the done thing but I don't like to have code lying around for weeks on end, not being merged because it's not on a release schedule.
Coding is a creative process, creativity cannot be planned. Imagine Van Gogh working according to a release plan.
Nick and Dave for bring Node-RED to live - amazing quality and flexibility and the entire Node-RED community.
Much thanks to
- @mwmiller for providing a fly server for running a live version of Erlang-Red,
- @joaohf many tips on coding Erlang and structuring an Erlang project, and
- @Maria-12648430 for debugging my initial attempt to create a gen_server for nodes.
No Artificial Intelligence was harmed in the creation of this codebase. This codebase is old skool search engine (ddg), stackoverflow, blog posts and RTFM technology.
AI contributions can be made according to the rules defined in .aiignore.