Support delaying jobs schedule with configurable delay time and start time #64
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Currently, Workflows and Jobs running by Helix requires more flexibility. For example, some of the jobs need to be started after some jobs finished for a certain mount of time. Same as Workflow, it may run at specific time, when some operations have been done. To better support Workflow and Job scheduling, Helix should provide a new feature to let user setup the delay time or starting for specific Workflows and Jobs. Workflows and Jobs should have an option that allow user set starting time of this Workflow or Job or set the delaying time for this Workflow and Job, when they are ready to start. Then Workflows and Jobs can be scheduled at correct time.
Purposed Design
The whole design has been split into two parts, generic rebalancer scheduling and delay time calculation. Since Job scheduling can be done via rerun WorkflowRebalancer, Workflow and Job delay scheduling can rely on the same generic scheduling mechanism. Generic task scheduling tasks the responsibiliy to set the running time for specific Workflow object. Then each object has its own starting time calculation algorithm.
Generic Task Scheduling
For generic task scheduling, it is better to have a centralized scheduler, RebalanceScheduler. It provides four public APIs:
public class RebalanceScheduler {
public void scheduleRebalance(HelixManager manager, String resource, long startTime);
}
Obviously, it offers schedule a rebalancer, get schedule time of a rebalancer and remove a rebalancer schedule. It also have an API that can invoke rebalancer immediately. With this RebalancerScheduler, each resource can be scheduled at certain start time.
Delay Time Calculation
Workflows have a property expiryTime, which is the delay time that for the Workflow. User can set it by call setExpiry method in WorkflowConfig. For Job, two methods, in JobConfig, will be provided: setExecutionStart and setExecutionDelay. Through these API, user can set the delay time and start time for Workflows and Jobs. Internally, Helix will take the delay time and start time, which is later.
For the logic implemented in computing Workflows and Jobs, Helix choose to do real time computation. User can set delay time or start time at JobConfig. When the job is ready to run, Helix will calculate the "start time" for delay via current time plus the delay time. Then compare it with start time if user set it up in JobConfig.
Inline image 1
Impact
From user perspective, user have to understand the difference between delay time and start time.
The WorkflowRebalancer will be called multiple times, which might be considered for performance.