Overseer is a workflow manager and a task scheduler. In overseer tasks are controlled by resources: tickets and flags. This feature makes it easy to create robust and flexible workflows. Overseer is inspired by Control-M from BMC.
Note that currently, overseer is in a demo stage; therefore, some parts of a project will change.
- Scheduling options: Manual, Daily, Day of a week, Selected months, Specific date
- Time criteria: Active from time, Active to time
- Hold / Free task / Rerun / Enforce
- Manual task confirm
- Task types: Dummy, OS,AWS (lambda,step function)
- Global variables, Task variables
- Flags
- Worker limits
- Cyclic tasks
- Management of an internal state of components, quiesce mode...
- Other task types: FTP, Messages, Services, Azure jobs, ...
- Logging
- Tasks running in the user's context
- Advanced relations between conditions
- Post Processing actions
- ...
git clone https://github.com/przebro/overseer
cd overseer
git checkout develop
make -f scripts/Makefile build
Inside def/samples directory, there are sample definitions that consist together for an example workflow. After a successful build, binaries can be found inside the bin catalog.
- ovs - scheduler, by default it starts listening on 127.0.0.1:7053
- ovsgate - api gateway 127.0.0.1:7054
- ovswork - worker, by default it starts listening on 127.0.0.1:7055
- chkprg - sample program
- tools/ovscli - tool, helps manage resources and tasks. There is also a web application client:(https://github.com/przebro/ovswc). It is recommended to use although it is also in a phase of a high development.
Task definition
Currently, tasks definitions are stored in catalog, by default it is def catalog located in a root directory of a project.
The basic task definition is:
{
"type" : "dummy",
"name" :"minimal",
"group" : "samples",
"schedule" :{"type" : "manual"}
}
type:Defines a kind of a task. Right now, there are two kinds of the task:
- dummy - an empty task that does not call any specific program or service but can have scheduling criteria and add/remove tickets for other tasks.
- os - executes scripts or programs on a worker.
- aws - executes lambda or step function name and group are unique identifiers of a task. Group represents a subfolder of a root of the definition catalog, and the name field represents the name of a json file.
schedule:Basically, there are two kinds of a task scheduling, manual and time-based. Manual means that task will be not taken into account by daily ordering process, any other type will be checked against specific, time-based criteria:
- daily: The task will be ordered everyday.
- weekday: The task will be ordered on a specific day of a week where 1 means Monday and 7 means Sunday.
- dayofmonth: The task will be ordered on a specific day of a month.
- fromend: The task will be ordered on a specific day from the end of a month, where 1 means the end of the month, 2 a day before the end, and so on. It is relative value so, fromend=2 in July will be resolved to 30 of July, and in February, it will be resolved to 27 of February or 28 if it is a leap year.
- exact: The task will be ordered exactly on a specific day: '2020-05-11','2021-04-07'...
Each of those values can also be restricted by specifying months in which the task can be ordered. The above example definition can be changed to:
{
"type" : "dummy",
"name" :"minimal",
"description" :"sample dummy task definition",
"group" : "samples",
"schedule" :{
"type" : "daily",
"months" :[1,3,7,12]
}
}
This task will be ordered every day of January, March, July, and December. Despite calendar-based criteria, every task can be restricted to run in a specific period. This can be achieved by setting two properties of schedule: from and to. For instance:
"schedule" :{
"type" : "daily",
"from" : "11:15",
"to" : "13:00",
"months" :[1,3,7,12]
}
These values restrict the time window when the task can run to hours between 11:15 and 13:00. A task can be capped at the bottom or the end so, setting only one of the mentioned values will make the time window half-open or half-closed.
Tickets are an essential element of tasks processing. They allow to define dependencies between tasks, use of tickets gives a posibility to build compound workflows. A task can require a ticket to run and, after complete their work, it can create a ticket for another task. More than one ticket can be expected by the task before starting the job. In that situation, a relation between input tickets can be defined. If relation is set to 'OR' then one of tickets is required and if relation is set to 'AND' then all of tickets are requiered. Here is an example:
"inticket" : [
{"name" : "IN-SAMPLE01A","odate" : "ODATE" },
{"name" : "IN-SAMPLE02A","odate" : "ODATE" },
],
"relation" :"AND"
A task with the above declaration can run only if these two tickets exist: IN-SAMPLE01A and IN-SAMPLE02A. Additionally, there is an order date parameter therefore tickets must have the same order date as a task. The "odate" parameter is resolved during the ordering process of a task and it is related to the task's calendar. This field can contain values like: "ODATE", "", "PREV", "NEXT","+nnn", "-nnn" where "nnn" is a number between 001 and 999. As mentioned, the calculated value depends on the task's schedule:
"schedule" :{
"type" : "daily",
}
...
"schedule" :{
"type" : "weekday",
"values" :[3,5,7]
}
If these definitions are ordered on the Friday 8 January 2021, the "ODATE" value will always resolve to the current date for both definitions but, "PREV" and "NEXT" will be different. For the first definition, "NEXT" and "PREV" will be resolved to Thursday and Saturday, but for the second definition "PREV" will resolve to Wednesday(3), and "NEXT" will resolve to Sunday(7). For this definition:
"schedule" :{
"type" : "fromend",
"values" :[1]
}
The task is ordered on the last day of a month, If the order date is 31 March, the "NEXT" will resolve to 30 April, and "PREV" will resolve to 28 February or, 29 February if it is a leap year.
There are two ways how the ticket can be added. Manually or, each task definition can contain a section that defines tickets that will be issued after successful completion of a task.
"outticket" :[
{"name" : "IN-SAMPLE01A","odate" : "ODATE" ,"action":"REM"},
{"name" : "IN-SAMPLE02A","odate" : "ODATE" ,"action":"ADD"},
{"name" : "IN-SAMPLE02B","odate" : "ODATE" ,"action":"ADD"},
]
If a task with that definition ends, the ticket "IN-SAMPLE01A" will be removed and, the ticket "IN-SAMPLE02A" and "IN-SAMPLE02B" will be added with an "ODATE" resolved to the order date. The Same rules as for an "inticket" definition applies to an "odate" field in an "outticket" definition so, "NEXT", "PREV" and, +nnn will resolve accordingly to the task's schedule definition.
Another type of resource is a flag. It helps manage interactions between tasks and prevents simultaneous execution of tasks that should not run together.
"flags" : [{"name" :"SMPL-FLAG","type" : "EXL"}]
When the task collects all tickets, it checks if it requires a flag to run; then, if a flag is defined for a given task, it checks if it can acquire a flag "SMPL-FLAG" if so, the task starts its execution. If any other task uses a flag with the same name, the task will back to the waiting state. A Flag can have one of two values:
- SHR: Shared means that multiple tasks can use a flag and run together.
- EXL: Exclusive, only one task can run with acquired flag of that name. For instance: if there are T1, T2 that have a flag with a type "SHR", and there is a T3 that has the flag with the same name but with a type "EXL", then at the same time, T1 and T2 can run but not T3, and if T3 runs, both T1 and T2 have to wait until T3 releases a flag. It is worth mentioning that the flag does not determine the priority of a task nor the order in which tasks can run. It prevents only a collision between tasks that cannot run together. There is a flag group inside the def catalog, which contains a sample chain of jobs that are controlled by flags.