Nodes and Edges Refactor

[stuartc]

This epic outlines the changes required to introduce the concept of edges and
nodes into Lightning. Re: #778

Currently our Triggers belong to Jobs and Workflows, this makes it difficult to
create a trigger as the workflow and the job must at least have an ID; and
ideally they should be persisted first.

We have some 'fancy' cast_assoc/3 functions that allow us to create a trigger
via a job that copies the jobs workflow_id to the trigger.

By introducing the concept of edges and nodes we can create a trigger without
associating it to a job; and vice-versa for a job.

Another use case is having conditions between node (jobs or triggers; but just
jobs for now), allowing for users to conditionally control the flow of their
workflows based on data.

Below represents a Job that has a condition on it, if the condition is true then
the subsequent job will be executed.

---
title: WorkflowEdge with Condition
---
flowchart LR
    A[Job A]-->|state.data == true|B[Job B]

---
title: WorkflowEdge with Runtime Condition
---
flowchart LR
    A[Job A]-->|on_success or on_failure|B[Job B]

ER Diagram

Given the ER diagram below, we can see that we have a WorkflowEdge model that
holds the relationship between a each node in the Workflow.

erDiagram
    Workflow {
        uuid id PK
    }
    WorkflowEdge }o--|| Workflow : belongs_to
    WorkflowEdge o|--o| Job : source
    WorkflowEdge o|--o| Job : target
    WorkflowEdge o|--o| Trigger : triggerable
    WorkflowEdge {
        uuid id PK
        uuid workflow_id FK
        uuid source_trigger_id FK
        uuid source_job_id FK
        string precondition
        uuid target_job_id FK
    }
    Trigger }o--|| Workflow : belongs_to
    Trigger {
        uuid id PK
        uuid workflow_id FK
    }
    Job }o--|| Workflow : belongs_to
    Job {
        uuid id PK
        uuid workflow_id FK
    }

The WorkflowEdge model has a precondition field that will hold an expression
that will be evaluated at runtime to determine if the edge should be followed.

*TDB: In order to represent "on success" and "on failure" conditions we
will need to represent these as true or state.error respectively. I have
concerns about leaning on the state for this since it's user modifiable, and
also get overridden in the case of a failure.

Triggers

Are no longer associated to a Job, but instead to a Workflow. This means that
there is no longer a need for a on_success and on_failure type on the model.

The result is that a Trigger becomes a Node that can be invoked from outside.
There will only be two kinds of Triggers for now, cron and webhook.

Plan

We should approach this with an additive approach, meaning we should add the new
features and columns and remove the old columns and features in a later commit
one we are able to:

1. Create a workflow, with triggers, jobs and edges.
2. Execute an Attempt via the Pipeline by using the new edges and nodes.

Model Changes

• Create a new WorkflowEdge model.
• Create a service to create a Workflow that creates the WorkflowEdges,
  Jobs and Triggers.
• Hunt down the places (starting with the test fixtures) where we create a
  Workflow, Job and Trigger and replace them with the new service.
• We may need to relax the database constraints on the Trigger model to allow
  for upstream_job_id to be blank.
• Triggers no longer require on_success and on_failure types.

The Edge model will have the following fields:

• workflow_id
• source_trigger_id
• source_job_id
• precondition
• target_job_id

LiveView Changes

• Create a new LiveView for creating a Workflow.
  This should allow a Workflow changeset to be built up by either messages from
  the ReactFlow UI or by the panel forms.

I believe an approach that could be beneficial for the LiveView is to create it
with much in the way of a UI at first, and then unit test the LiveView with
event messages and assert the changeset is correct.

These tests should be either changed or thrown out as the UI is added, but it's
important we get the message handling correct from the beginning. We want to
make sure that (for example) a new Job can be added via ReactFlow, and the form
panel will update the changeset correctly.

{
  workflow: {
    edges: [{
      source_trigger_id: "abcde...",
      target_job_id: "<<new generated uuid>>",
    }],
    job_nodes: [{id: "<<new generated uuid>>"}],
    trigger_nodes: [{id: "abcde...", type: "cron", cron_expression: "..."}]
  }
}

In the above example we have some params that will be passed to Ecto. We need to
pay special attention to the edge record, when the newly created job is invalid,
we need to make sure we don't get spurious errors when the edge is built.

Non-exaustive list of things to support:

• Allow a new Node to be inserted into ReactFlow and params are sent to the
  LiveView
• Allow Edges to be created via drag and drop
• Allow Edges to be created with a Node
• Add a Trigger panel
• Removing a Node while the panel is open should close the panel
• Integrate existing Job panel (without trigger)

ReactFlow Changes

• Create a new ReactFlow component that allows for the creation of a Workflow.
• Allow for the creation of a new Node (Job or Trigger)
• Allow for the creation of a new Edge
• Trigger the opening of the panel when a Node is clicked

The component will receive a data structure that represents the Workflow with
its Jobs, Triggers and Edges. See JSON example above

The outer component (the one that is rendered by the LiveView) will be
responsible for sending data to ReactFlow and handling events from ReactFlow
when a Node or Edge is added or modified.

The outer component will also be responsible for sending data to the LiveView
when a Node or Edge is added or modified.

Integration

Since we are opting for a more additive approach, we will have an issue that
will represent the work involved with removing the old features and columns and
constraints.

[josephjclark]

I'm coming at this with very little understanding of Lightning's current architecture (let alone Platform's), but I do have some observations and inputs.

I don't think the idea of a Trigger exists in runtime/runtime manager/CLI land. I think it's purely an artefact of Lightning.

We can treat the Trigger as a special type of Node with no expression (a jobless node) and one or more edges. And actually I think this is exactly how the Execution Plan should model a trigger.

Each Edge should have exactly one source and exactly one target, and may have a condition (not a precondition).

And of course any node (including triggers) can have multiple associated Edges, which is implied in the Edge FKs.

So I think your entities are more like:

WorkflowEdge
id
workflow_id
source_id
target_id
condition

WorkflowNode
id
workflow_id
expression

Maybe you want a type job | trigger on a node, or a trigger field with a value of cron | `webhook¬

[stuartc]

@josephjclark You're absolutely right, I believe I did state that triggers will only be cron and webhook nodes, and rightly so are only Lightning concerns currently.

For v1 of the execution plan, I'm expecting to exclude trigger nodes and edges with them as sources.

Am I missing anything?

Edit: I'll give merging Triggers and Job a thought 😅