A go task scheduler / runner library with various backends and triggers
Explore the docs »
Report Bug
·
Request Feature
Table of Contents
There are a handful of golang task schedulers but none of them suited our needs so we created one. Our initial need was for reminders (texts/emails/sms) with various triggers like cron, execute once, or natural language such as "every other tuesday"
Reasons we rolled our own scheduler:
- We wanted a trigger interface so that new triggers can easily be created
- We wanted a storage interface so that the scheduler can be run in different modes for different scales
Distinct Features:
- Trigger interface (this isn't unique to this project)
- Support for any storage backend you want to create
- Horizontal scalability
- Configurable execution window to control resource usage
Implemented Backends:
- Cockroachdb - To allow horizontal scalability, useful for real-world production scenarios
Planned Backends:
- Memory - An in-memory backend, useful for toys/testing
Implemented Triggers:
- ExecuteOnce - Executes once at the specified time, respects retries and expiration
- Cron - Executes on a cron schedule, respects retries and expiration
Planned Triggers:
- Natural language - Execute every other tuesday at 9:34am
FAQ:
- What do you mean by "execution window"
- The execution window is a
time.Duration
that you pass to theNewScheduler()
function. This defines how often to fetch tasks from the storage backend. This is configurable mostly to control resources. For example if set to 30 seconds, then every 30 seconds the scheduler will get the tasks scheduled to execute in the next 30 seconds and hold them in memory, executing your handler at the task's scheduled time. You can tune this to control memory usage.
- The execution window is a
- What does the
ExpireAfter
field on a task do?- This setting is used for fault tolerance. The backend store tracks when a task is in progress. If a task's scheduled time is in the past, the store will re-schedule the task if the
ExpireAfter
has passed. This would happen if there was some failure to update the task in the store, or if the handler hung, or something like that so that the task doesn't just get dropped. This lets you have handler functions that run longer than the execution window without executing multiple times.
- This setting is used for fault tolerance. The backend store tracks when a task is in progress. If a task's scheduled time is in the past, the store will re-schedule the task if the
Design: The scheduler runs 3 goroutines on tickers that each have distinct concerns but don't care about each other. This design is intended to ease troubleshooting, implementation, and maintenance by avoiding complex logic through separation of concerns.
- Scheduler routine queries for task definitions that should run in the next window, and creates task instances accordingly. When a task instance is created, the scheduler also updates the task definition's
next_fire_time
based on the definition's trigger - Task runner routine queries for task instances that should run in the next window and haven't run yet, or are expired, and runs them at their specified time. The runner also sets the task instance's
started_at
time when the handler is called, and sets thecompleted_at
time when the handler is finished, if the handler is successful - Cleanup routine deletes completed task instances and task definitions if appropriate.
This example uses the cockroachdb store, ran via gnomock, you can use any store that exists or PR your own.
// start a gnomock cockroach container
dbName := "test"
cockroachdbContainer, err := gnomock.Start(cockroachdb.Preset(cockroachdb.WithDatabase(dbName)))
uri := fmt.Sprintf("host=%s port=%d dbname=%s user=%s password=%s sslmode=disable",cockroachdbContainer.Host, cockroachdbContainer.DefaultPort(), dbName, "root", "")
// instantiate a new store
cockroachdbStore := cockroachdb_store.NewCockroachdbStore(uri, nil)
// define a handler function
handler := func(task pkg.Task) error {
fmt.Println("handling a task")
return nil
}
// instantiate a scheduler instance, defining the window to get scheduled tasks
scheduler, err := pkg.NewScheduler(1*time.Second, handler, store)
// start the scheduler in a goroutine
go scheduler.Run()
// define a task
id := uuid.New()
metaData := map[string]string{"some metadata": "about the task, this can be whatever you want"}
executeAt := time.Now().Add(5 * time.Second)
expireAfter := 10 * time.Second
task := pkg.Task{
Id: &id,
Metadata: metaData,
ExpireAfter: &expireAfter,
ExecuteOnceTrigger: pkg.NewExecuteOnceTrigger(executeAt),
}
// schedule the task
err := scheduler.ScheduleTask(task)
// your handler will now be called at the time specified in the trigger, and the task you defined will be pased to your handler
There are no prerequisites, it's an importable go library. If you're using a backend that requires a database, you have to deploy that database somewhere.
go get github.com/catalystcommunity/go-scheduler
- Add Cockroachdb backend support
- Add Natural language trigger
See the open issues for a full list of proposed features (and known issues).
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature
) - Commit your Changes (
git commit -m 'Add some AmazingFeature'
) - Push to the Branch (
git push origin feature/AmazingFeature
) - Open a Pull Request
Distributed under the APACHE 2.0 License. See LICENSE.txt
for more information.