Persist Scheduled Tasks to the Backing Store

[brthor]

Currently, when a Gofer.NET worker resets, it's log of scheduled tasks that need to be accounted for, and run on their specified schedules is also reset.

Scheduled tasks are, in fact, specific to the worker they are created on. Although logic exists to prevent scheduled tasks with the same key from having duplicate runs across multiple workers, no method exists to persist and share scheduled tasks across workers.

Multiple requirements:

1. When a worker restarts, or is brought up, it should load the current set of scheduled tasks into it's memory, and account for them in it's scheduler loop.

2. When a new scheduled task is added to any worker it should be persisted to the backing store.

3. When a new scheduled task is added to any worker sharing the same backing store, that scheduled task should be loaded into each of the other workers memory as a part of their scheduler loop, although not necessarily in every iteration. Some small amount of lag between a task being added to the store and existing workers picking it up is acceptable.

[brthor]

This issue is more important than I realized.

Currently, a common pattern is to have a web client offloading work to a set of workers.
The case where a web client schedules tasks, but does not also listen for tasks causes the scheduled tasks never to be run.

[brthor]

In solving this issue, an important design consideration came up regarding recurring tasks.

I'm writing about it here to record the tradeoffs in case the consideration should be revisited.

In the event of all task schedulers failing or being turned off, and then being turned on after some non-negligible amount of time, the consideration is whether or not the recurring tasks should run for any missed intervals during that time off.

What was implemented does not run missed intervals for recurring tasks but will start running the next intervals as soon as at least one scheduler comes back up.

The motivating scenario was if the schedulers are manually turned off for an evening (or so) then when they get switched back on for the next day, a flood of recurring tasks should not then be poured into the task queue.

An important tradeoff here is that a task scheduler that is backed up may skip recurring task intervals.