Skip to content

Shepherd Architecture[Part#2]

Jump to bottom
devsda edited this page May 29, 2019 · 4 revisions

Introduction

Shepherd is a highly scalable workflow management Engine to create, schedule and monitor workflows programmatically in any language. It can support conditional as well as un-conditional workflows. It is a Software as a service, where clients can register themselves, create multiple endpoints, and execute workflows in any programming language, or can use multiple programming languages in a single workflow.

Sequence diagrams

Client registration workflow

  1. This is the very first step. This is the API through which client can register himself in Shepherd.
  2. While registering, client has to pass “clientName”. [Currently, skipping security discussion. Will prioritise in V2 cut.]

Endpoint creation workflow

  1. This API helps to create new endpoint. There can be multiple endpoints against single client. For example, for client -> “data_platform”, possible endpoints -> “data_platform_beta”, “data_platform_staging”, “data_platform_prod”, “hitesh_dev_data_platform”, etc.
  2. To setup endpoint, client has to share workflow graph (in xml format), and corresponding endpoint details (in son format).

Execute workflow

  1. This API helps to execute workflow.
  2. For executing workflow, client has to pass clientName, endpoint, initial payload. Shepherd will return objectId (Identifier of workflow in Shepherd).
  3. Parameters to call, clientName, endpoint, and initial payload.

Node Executor Workflow

Node Tracker Workflow

Resume workflow

  1. This helps to resume workflow from where it failed last time.
  2. When Shepherd gets request to execute workflow, it generates objectId for the same. Also, it creates execution_id, which is the actual running instance of given objectId (client request).
  3. One ObjectId can contain multiple executionIds. It depends upon how many times client requested for restart operation.
  4. Parameters : ObjectId

Restart workflow

  1. This API helps to restart workflow, This means, it kills the current running instance of given objectId, if there is any and start a new running instance.
  2. By this operation, it increases the total execution counts by 1 for given object id.
  3. Parameters : ObjectId

Kill workflow

  1. This API helps to kill the workflow. This means, it kills the current running instance of given objectId, if there is any.
  2. Parameters : ObjectId

Get status

  1. This API helps to get status of given workflow. Possible status -> PROCESSING, COMPLETED, FAILED
  2. Parameters : Object

References

Clone this wiki locally