Shepherd Architecture[Part#1]
This wiki explains architecture of Shepherd in detail.
Shepherd is a highly scalable workflow management Engine to create, schedule and monitor workflows programmatically in any language. It can support conditional as well as un-conditional workflows. It is a Software as a service, where clients can register themselves, create multiple endpoints, and execute workflows in any programming language, or can use multiple programming languages in a single workflow.
- Workflow - Workflow is a directed acyclic graph, which consists of Nodes, Edges.
- Node - Node is the fundamental unit of graph. We can consider it as single processing unit. Two nodes can be connected through edge.
- Edge - Edge is a connecting bridge between two nodes.
- Object - Processing request corresponding to a given endpoint. It can be in any state -> PROCESSING/KILLED/COMPLETED/FAILED.
- Execution - Execution is the actual running instance of object. It can be in any state -> PROCESSING/KILLED/COMPLETED/FAILED.
- Node-Message - Node-Message is a packet that stores in RabbitMQ. It contains all the information that is required to execute given node on Node-Executor/Node-Tracker.
.png)
Below are the major components in the whole Shepherd Architecture -
- Shepherd-Core
- Node-Executor
- Node-Tracker
- RabbitMQ
- MongoDB
- SQL Database
In this section, we will cover the responsibilities played by major components.
Shepherd-Core is the web-server, which serves requests coming from Shepherd-Web/Rest-API. This is the entry-point component in Shepherd-backend, which means it plays a request management role as well. This component serves few requests in a sync way, for example : Client Registration, Endpoint Registration, Update endpoints, Retrieve state of given endpoint execution. And, others in async way, for example : Execute endpoint. For this use-case, it pushes Node-Message to RabbitMQ, which later served by Node-Executor.
Node-Executor is a Node-Messages Consumer running over RabbitMQ. This component consumes messages and process all fetched nodes. It uses "Ready to Execute" Queue(also known as Primary Queue) for consuming node-messages.
Execution of Node includes below steps :
- Execute Client API.
- Update State of Node, and execution in MSSQL.
- Find out next(children nodes) ready to execute nodes.
- Push ready to execute nodes into Primary Queue(Ready to Execute Queue), and remaining in Secondary Queue Secondary Queue is used to avoid any Node to become Zombie in execution race condition.
Node-Tracker is a Node-Messages Consumer running over RabbitMQ. It uses "Not ready to execute" Queue(also known as Secondary Queue) for consuming node-messages. This helps Shepherd to avoid making any node in Zombie state. Eventually, it saves Execution from entering into Zombie state. The whole responsibility of this component to promote node-message from secondary to primary Queue once its ready to get executed.
Shepherd is using RabbitMQ for all its pub-sub use-cases. Shepherd maintains 2 queues in RabbitMQ : "Ready to execute Queue"(also known as Primary Queue), and "Not Ready to execute Queue"(also known as Secondary Queue). Shepherd-core is pushing node-messages in primary Queue whenever it gets execute workflow request, Node-Executor is pushing node-messages in Secondary Queue, when currently touched node is not ready to process, and Node-tracker is promoting node-message from secondary to primary Queue, once its ready to get executed or discarding the node-message, if its in processing state already by some other process. This helps Shepherd to avoid any duplicity at Node level.
Shepherd is using MongoDB for sharing data/information between nodes under execution. With this responsibility, we can also call it "Inter Node Communication" component. This helps any node to put any data, which later on used by any subsequent node. This document will be available in DB for 1 month, which can be used for auditing purpose. After a month, Shepherd will hard delete that document. Customer will get 2 MB space per execution.(Restriction by MongoDB) It's a recommendation to share only metadata through given document.
Shepherd is using SQL database to store state of Execution, and Node. It also maintains client, and endpoint details, which can be retrieved through Shepherd-web or Rest-APIs. Currently, Shepherd is using MySQL engine through JDBI. To know about current database schema in detail, please check : Shepherd Database Schema