Capability In Conductor To Schedule Workflows

[jas34]

Problem statament:

1. At times there are use cases in which we need to run some tasks/jobs only at a scheduled time.
2. In microservice architecture maintaining schedulers in various microservices is a pain.
3. We should have a central dedicate service that can do scheduling for us and provide a trigger to a microservices at expected time.

Solution:

1. Conductor being service orchestrator already working as a central service in a microservice-based landscape.
2. It can be enhanced with scheduling capability to start a workflow at a scheduled time and task then can be executed in microservice at expected time.
3. Workflow engines like apache airflow already provide such capabilities but apache airflow is not meant for service orchestration hence, not a right fit as a solution to the problem.

Suggested Runtime Model:
A new module for workflow scheduler can be added to the conductor. This module can be enabled on-demand like ArchivingWorkflowModule.

Component Low-Level Details:
The following diagram shows how workflow scheduler module looks like and how it can start workflow at a scheduled time?

1. Scheduler Manager:
   • This component is meant to manage lifecycle of scheduled workflow.
   • It takes lifecycle state decisions for a scheduler with the help of a scheduled jobs registry.
   • It schedules the schedulers through a scheduling assistant.
2. Scheduled Jobs Registry:
   • This registry acts as a single source of truth to know whether a particular workflow is required to be scheduled, paused, or deleted.
3. Scheduling Assistant:
   • This is an abstract layer for job scheduling.
   • It exposes methods to:
     - creates a job.
     - pause a job.
     - delete a scheduled job.
4. Scheduled Jobs:
   • These jobs are scheduled by scheduler manager and are suppose to run at scheduled time.
   • Each scheduled job is provided an instance of Runnable task which starts a workflow at a scheduled time by using com.netflix.conductor.service.WorkflowService
5. LockingService:
   • In order to ensure that in a clustered environment workflow is triggered only once at a scheduled time we can have locking service built on the top of com.netflix.conductor.service.ExecutionLockService

@apanicker-nflx / @aravindanr let me know your thoughts on this. If the idea looks justified we can get into more technical details.

[maheshyaddanapudi]

Please have a look at API Calls Scheduler or Conductor Workflow Scheduler

They are both based on same Spring Boot + Quartz + MySQL design and are well tested against Conductor Workflows SCHEDULED executions.

If agreed, its just a matter of simple merge and exposing extra APIs. Though is easier said than done, as current the API Scheduler supports H2 / MySQL only. It can be easily made to support Postgres. However, I am unsure about Redis. Still it can be explored as most of base code would be readily available.

[jas34]

@maheshyaddanapudi thanks for showing interest in the idea and sharing valuable feedback.
I have gone through suggested projects. Lets me clarify the view of idea again:

• The above idea is to enhance conductor server itself with a new module for scheduling workflows. This module is expected to run within conductor server when enabled through property conductor.additional.modules.
• The base work on this as an MVP project available in Schedule Conductor Workflows.
• The project follows the core design principles of conductor. This has been built as an extension module of conductor and it does not bring with it any specific dependency like quartz with it. (Feel free to open ticket in the project for any further clarification).
• On suggested projects:
  • API Calls Scheduler:
    • This is a REST interface over quartz. Great work!
    • It does not fits into the idea because it is hosted as a separate service that is tightly bound to relational databases because of quartz.
  • Conductor Workflow Scheduler:
    • This is also REST client of conductor server.
    • It starts conductor workflow at a scheduled time through REST trigger ScheduleConductorWorkflowExecutionJob
    • It does not fits into idea because it falls into the same design where one will maintain two services (scheduling and conductor) which is tightly coupled with Quartz.

[jas34]

@apanicker-nflx / @aravindanr may I have some feedback on idea?

[apanicker-nflx]

@jas34 This is a very interesting solution to the scheduling use-case that users may be looking to solve using Conductor. There is a definite advantage in re-using Conductor's persistence and locking layers and not having to depend on another external system to meet the scheduling requirements.

Please create a PR to link to your project from the related community projects page so that interested users can avail of this functionality and also contribute/help maintain this addition.

[jas34]

@apanicker-nflx thanks for the review. I will come back with PR soon.

[jas34]

@apanicker-nflx as per your suggestion created PR: #2356

Any plan of including https://github.com/jas34/scheduledwf module 'scheduledwf-module' in Netflix/conductor repo?

[apanicker-nflx]

Thanks for linking to your contribution and making it available to the community.

Any plan of including https://github.com/jas34/scheduledwf module 'scheduledwf-module' in Netflix/conductor repo?

Unfortunately, since we cannot directly support/maintain this contribution, we would not be able to add this to the main Conductor project. However, we would recommend you to continue to support this module and build traction in the community around this contribution. Thanks

[jas34]

@apanicker-nflx thanks for the answer.
I have updated the answer to a question on scheduling workflow in faq.md. PR for the same is here #2373