Workflow Automation System

This project provides a system for automating business processes through workflows. It allows users to create and manage workflows and enables developers to extend the system via a plug-in model.

Disclaimer: This project was initially developed using proprietary internal libraries. To prevent exposure of usage and features of these libraries, which were protected by a Non-Disclosure Agreement, the Git history has been reset. Parts of the backend were rewritten from an OData/Azure Cosmos DB stack to regular REST APIs with EF Core and PostgresDB. The frontend, originally in Blazor, has been rewritten to SvelteKit using AI tools. Original contributors are @paulbujor, @alexandrutatarciuc, and @ranteidz. The project itself (code, findings, results) are the contributors's intellectual property.

Project Goals

• Workflow Management GUI: Develop a graphical user interface for creating, managing, and monitoring workflows.
• Automation Services: Implement services to execute workflow steps when triggered.
• Plug-in Model: Create an SDK for developers to integrate new components, with automatic UI generation.

Key Concepts

• Workflow System: The platform for workflow creation and execution.
• Workflow SDK: A plug-in model for integrating external applications and custom components.
• Workflow: A sequence of steps executed in order, triggered by an event.
• Application: An external system (e.g., email clients, CRM) integrated into the Workflow System, exposing Workflow Components.
• Workflow Component: A modular unit within a Workflow, either a Trigger or an Action. Includes inputs, outputs, and authentication data.
• Workflow Action: A Workflow Component that performs an operation within an integrated Application.
• Workflow Trigger: A Workflow Component that initiates Workflow execution based on an event in an integrated Application.

System Architecture

The system uses a service-oriented architecture with microservices and serverless computing for extensibility and scalability.

Core Services

• Web Client: Frontend GUI for workflow creation and management.
• Workflow Manager: Backend service for workflow persistence and management.
• Component Registry: Backend service for discovering and exposing Workflow Components from integrated Applications.
• Orchestrator: Serverless functions for scalable workflow execution.

Workflow Execution Flow

The Orchestrator is central to workflow execution, designed for high scalability using serverless functions. The process involves several key components:

• Trigger Handler: Receives incoming trigger events, typically via HTTP calls, and places the event payload into a queue for asynchronous processing.
• Trigger Handler Processor: Monitors the queue. Upon detecting new items, it dequeues them, identifies all workflows configured to be triggered by that event (via the Workflow Manager API), and initiates a Workflow Resolver instance for each.
• Workflow Resolver: Takes an Orchestration Request (containing workflow and trigger data). It retrieves the full definitions of all Workflow Components involved in the workflow from the Component Registry. These definitions are then used to construct an Orchestration Execution object, which is passed to the main Orchestrator for execution. Component retrieval is parallelized for efficiency.
• Workflow Orchestrator (Execution): Executes the steps of a workflow. It currently executes Workflow Actions in sequential order. For long-running workflows, it leverages Durable Functions to manage state and execution without billing for idle time.
• Variable Handling: The system supports passing outputs from one Workflow Component as inputs to subsequent components, using a variable referencing system. This allows for complex data flow within workflows.

Component Discovery and Integration

The system's plug-in model relies on applications exposing their Workflow Components for discovery and integration.

• Application Integration: Applications integrate with the Workflow System by using the Integration SDK. This typically involves a few lines of code in the application's startup, such as registering components (Actions and Triggers) that the application exposes.
• Component Registry's Role: The Component Registry service is responsible for discovering and persisting the definitions of these Workflow Components. It communicates with integrated applications via a discovery URL to retrieve metadata (e.g., name, description, input/output schemas) for each exposed component.
• Automatic UI Generation: The extracted metadata allows the Workflow System to automatically generate the necessary user interface elements for configuring and using these components within the Web Client, without requiring manual UI development for each new component.
• Dynamic Availability: Once discovered and registered, these components become dynamically available for users to incorporate into their workflows.

Workflow SDK

The SDK enables system extension and includes:

• Actions SDK: For creating Workflow Action components.
• Triggers SDK: For creating Workflow Trigger components.
• Integration SDK: For integrating Applications and their components, handling setup and metadata.

Technology Stack

• Frontend: Svelte, Vite, TypeScript.
• Backend: .NET (C#), ASP.NET Core.
• Containerization: Docker, Docker Compose.
• Infrastructure: Infrastructure as Code (Bicep).
• CI/CD: YAML pipelines for Azure DevOps.
• Data Storage: PostgresSQL databases.

Project Structure

• Apps/: Example plug-in applications.
• Backend/: Core backend microservices (Component Registry, Orchestrator, Workflow Manager, shared libraries).
• Frontend/: Web Client source code.
• Infrastructure/: Cloud infrastructure definitions.
• Pipelines/: CI/CD workflow definitions.
• Sdk/: Workflow SDKs.

Component Usage Examples

This section provides examples of how Workflow Components are defined, implemented, and integrated within an application using the Workflow SDK.

Defining a Workflow Trigger

Workflow Triggers are defined by inheriting from WorkflowTrigger<TOutput>, where TOutput specifies the data payload associated with the trigger. Below is an example of a NewOrderTrigger:

using Workflow.Sdk.Triggers;

namespace Workflow.Apps.OnlineStore.Backend.Components;

public class NewOrderTrigger(IHttpClientFactory clientFactory) : WorkflowTrigger<WfOnlieOrder>(clientFactory)
{
    public override Guid Key => new("e208988f-8c38-4d73-a47c-999311552f75");
    public override string? Name => "New Order Received";
    public override string? Description => "Triggered when there is a new order";
}

Calling a Workflow Trigger

Application developers call Workflow Triggers from their codebase when a specific event occurs. The SDK handles sending the trigger event to the Workflow Orchestration services. For example, to trigger the NewOrderTrigger with a WfOnlieOrder payload:

// Assuming 'newOrderTrigger' is an instance of NewOrderTrigger
newOrderTrigger.TriggerAsync(new WfOnlieOrder { /* ... order data ... */ });

Implementing a Workflow Action

Workflow Actions are defined by inheriting from WorkflowAction<TInput, TOutput> (or its variations like WorkflowActionInputOnly<TInput>). The core logic resides in the ActionAsync method. Below is an example of a SendEmailAction:

using System.Net.Mail;
using Workflow.Apps.Smtp.Services;
using Workflow.Apps.Smtp.Workflow.Models;
using Workflow.Sdk.Actions;
using Workflow.Sdk.Components.Attributes;

namespace Workflow.Apps.Smtp.Workflow.Actions;

[ActionRoute("SendEmail")]
public class SendEmailAction(ISmtpClientService client) : WorkflowActionInputOnly<Email>
{
    public override Guid Key { get; } = Guid.Parse("9df93a52-89a7-4485-a217-9eac924ccc26");
    public override string Name => "Send Email";
    public override string Description => "Sends an email from an Outlook SMTP hardcoded address";

    public override async Task ActionAsync(Email input)
    {
        client.SendEmail(new MailMessage("notification@workflow.com", input.Recipient, input.Subject, input.Body));
        await Task.CompletedTask;
    }
}

Registering Components in an Application

Applications register their Workflow Components during startup, typically in Program.cs (or Startup.cs in older .NET versions), using the SetupWorkflowIntegration and AddComponent methods provided by the Integration SDK. This makes the components discoverable by the Workflow System.

// Add Workflow Components
builder.Services.SetupWorkflowIntegration(baseUri ?? string.Empty)
    .AddComponent<NewOrderTrigger>();
    .AddComponent<SendEmailAction>(); // Example for an Action

// ... other startup code ...

app.AddWorkflowDefinitions();

This setup ensures that registered components are exposed and their metadata is available for the Workflow System to use, enabling their selection and configuration within the Web Client.