Repository navigation
v1.3.0-beta.2
Pre-release
Pre-release
A prerelease. It makes the end of a job reliable when the store fails at the wrong moment, and stops the Redis heartbeat from starting running jobs a second time. No public API change and no database schema change since 1.3.0-beta.1.
Fixed
- A job's outcome is no longer lost when its final write fails (#74, #80). The job leaves the list of running jobs on every path, so a stop no longer waits for it and its cron's next occurrence is no longer skipped. The scheduler writes the outcome again while the instance runs, at most 5 seconds apart, until the store takes it; while outcomes wait it logs an Error from 5 minutes and a Critical from 10. Chain children are run, handed off or skipped only once the parent's outcome is stored.
- A late write of an old run no longer overwrites a newer run (#74). Every write of a running job is fenced by the marker its claim wrote. The stop's
Cancelledwrite is fenced too and no longer replaces a stored outcome. - Saving a whole time ticker no longer runs it again (#56).
UpdateAsyncand the dashboard edit keep the run state of a row that is running or has ended. - Redis: a live node's running jobs are no longer released after a Redis outage (#107). A node whose heartbeat is missing is released only when it stayed missing for one more heartbeat TTL (
NodeHeartbeatInterval+ 20 s), and a node never releases itself. Before, after an outage longer than the TTL, the first node back released the others' running jobs and they started a second time. - Redis: a node that starts again under the same name releases what its earlier run left locked (#109). Before, those jobs stayed
InProgressand never ran again; a single Redis instance had no release at all. - A put-back that failed inside a claim call is retried (#100). A job could stay
InProgresswith no worker until the node restarted. - Code-defined crons (first step of #92). Redis no longer fails to start when a function has two crons. EF Core and MongoDB no longer overwrite a user's cron of the same function: only the seeded rows take the expression in code.
- A job waiting for a concurrency slot no longer starts while the application stops (#72); it goes back to
Idle.
Behaviour changes to check before upgrading
UpdatedAtof a running job holds the claim marker for the whole run; it no longer moves with every write of the run.- An ended job is not run again by saving it. Add a new job instead.
- Redis releases a dead node later: about two heartbeat TTLs after its last heartbeat, not one.
- Redis, one name per running instance. Two processes that run at once under one
NodeIdentifiernow release each other's running jobs at startup, as with EF Core. Give each instance its own name. - Redis writes are bounded. A row that keeps changing under a write is retried 5 times, then the write throws.
- Writes during a run (a failed attempt, the next retry number) are best effort: a failure is logged as a Warning and no longer fails the job.
- Several crons of one function, none marked as seeded: the seed changes none of them and logs a Warning at every start.
- Custom persistence providers are not fenced: they keep the behaviour of 1.3.0-beta.1 for all of the above.
Known limits in this prerelease
- Inline chain children after an outcome that was stored late. When a parent's final write fails and the scheduler stores it later, the parent's children without a time of their own are neither run nor skipped; they stay
Idle. The log names the parent. Delayed children are not affected. This is the remaining step of #74 and will be fixed before 1.3.0. - An inline chain child whose node was killed is released to
Idleand not run again (#84). - At-least-once. A job whose process ended before its lost outcome was stored can run again after its rows are released.
- Redis: a node that alone could not reach Redis for about two heartbeat TTLs, while the others could, has its running jobs released (#46).
- Code-defined crons: replicas that start together can still create the same cron twice (#92).
- A job can start up to one second before its time when a cron occurrence is due in the same second.
Corrections to the 1.3.0-beta.1 notes
- "Jobs start on the scheduler's own worker threads" (#61) holds for functions without a request type. A function with a typed request still starts its code on the .NET thread pool.
- The put-back of a claim whose job never starts (#44, #54) was complete only for a put-back that succeeds at the first try; the retry of a failed put-back comes with this prerelease (#100).
- A delayed chain child "never starts before its own time": it can start up to one second early (see above).
What's Changed
- Do not start a claimed job whose slot came free as the application stopped by @msx752 in #72
- Retry a put-back that failed inside a claim call by @msx752 in #100
- Add container scenario tests for the fixes on develop by @msx752 in #101
- Bring only the seeded crons to the expression in code (first step of #92) by @msx752 in #102
- Keep a job's end consistent when its outcome cannot be stored by @msx752 in #103
- Fence the writes of a running job by its claim marker by @msx752 in #104
- Write a lost outcome again and fence the stop's Cancelled write by @msx752 in #105
- Add container scenarios for #74 and the first step of #92 by @msx752 in #106
- Hold the worker in the priority tests with a blocking job by @msx752 in #108
- Release what an earlier run left locked when a Redis node starts by @msx752 in #109
- Release a Redis node only after its heartbeat stayed missing for a TTL by @msx752 in #107
- Add a container scenario for the Redis heartbeat after an outage by @msx752 in #110
- Correct five statements of the guide that said more than the code does by @msx752 in #111
- Release 1.3.0-beta.2 by @msx752 in #112
Full Changelog: v1.3.0-beta.1...v1.3.0-beta.2