Repository navigation
1.3.0
1.3.0 changes how chains run on every store and fixes several ways a job could run twice, never run, start early or stay InProgress. It is the code of 1.3.0-beta.3. Please read Upgrade before deploying. No database migration for EF Core and no change of the MongoDB collections; Redis gets new keys (see Redis).
Delayed chain children (#50)
A child or grandchild with an execution time of its own (SetExecutionTime) now waits for its parent on every store: EF Core, MongoDB, Redis and in-memory.
- It runs only after its parent ended with a status its
RunConditionaccepts, and not before its own time. - If that time has already passed, it starts right after the parent and ends
Done. One started by the fallback endsDueDone. - When the parent ends with a status the condition does not accept, the child is recorded
Skipped, with everything below it.
Before, a delayed child ran right after its parent and its own time was ignored; on EF Core and MongoDB it was handed out at its own time, without waiting for its parent.
Fixed
Jobs that ran twice
- A due job runs once, on the node that won its claim (#45). The scheduler loop and the fallback could both run the same row, on several nodes or on one.
- A late write of an old run no longer overwrites a newer run (#74). Every write of a running job is fenced by the marker its claim wrote.
- Saving a whole time ticker no longer runs it again (#56).
UpdateAsyncand the dashboard edit keep the run state of a row that is running or has ended. - Redis: a live node's running jobs are no longer released after a Redis outage (#107). A node whose heartbeat is missing is released only when it stayed missing for one more heartbeat TTL (
NodeHeartbeatInterval+ 20 s), and a node never releases itself.
Jobs that never ran or stayed InProgress
- A job's outcome is no longer lost when its final write fails (#74, #80). The scheduler writes the outcome again while the instance runs, at most 5 seconds apart, until the store takes it; while outcomes wait it logs an Error from 5 minutes and a Critical from 10. A stop no longer waits for such a job and its cron's next occurrence is no longer skipped.
- Chain children follow a parent outcome that was stored late (#74). Those it lets run are claimed from the store and run, those it does not let run are skipped with their descendants.
- A claim whose job never starts is put back (#44, #72, #100). This covers an application that stops while the job waits for a slot, a frozen or refusing scheduler, and a claim call that failed after its write landed.
- Redis: a node that starts again under the same name releases what its earlier run left locked (#109).
- MongoDB: "Run now" on a cron works (#52), and grandchildren of a chain are dispatched (#51).
Other
- A job no longer starts before its time when another job is due in the same second (#117). It could start up to one second early.
- Jobs of functions without a request type start on the scheduler's own worker threads (#61), so a busy thread pool no longer delays them. A function with a typed request still starts its code on the .NET thread pool.
- Code-defined crons (first step of #92). Redis no longer fails to start when a function has two crons. EF Core and MongoDB no longer overwrite a user's cron of the same function: only the seeded rows take the expression in code.
Behaviour changes to check before upgrading
- Rows stored before the upgrade follow the new chain rules on EF Core and MongoDB. A pending delayed child whose parent has not ended waits. One whose parent already ended without a match is recorded
Skipped. - An inline child (one without a time of its own) is recorded
InProgresswhile it runs (#68). This costs one store write per inline child before it starts. - A child that follows a late-stored outcome runs as a job of its own: it takes its own concurrency slot and priority.
UpdatedAtof a running job holds the claim marker for the whole run; it no longer moves with every write of the run.- An ended job is not run again by saving it. Add a new job instead.
- Writes during a run (a failed attempt, the next retry number) are best effort: a failure is logged as a Warning and no longer fails the job.
- Several crons of one function, none marked as seeded: the seed changes none of them and logs a Warning at every start.
- Filters on
DueDonesee fewer rows. - Custom persistence providers are not fenced and do not follow late-stored outcomes; for those points they keep the behaviour of 1.2.3.
Redis
- Each chain child is a record of its own, and the dashboard shows its real status (#69). New keys:
tq:tt:{id}per child, plustq:tt:{id}:children. - Chains stored by 1.2.3 or earlier keep their children nested in the parent's record and keep the old behaviour: a delayed child runs right after its parent. The dashboard does not show those children's results, and such a child cannot be deleted on its own; delete the chain.
- A dead node is released later: about two heartbeat TTLs after its last heartbeat, not one.
- One name per running instance. Two processes that run at once under one
NodeIdentifierrelease each other's running jobs at startup, as with EF Core. - Writes are bounded. A row that keeps changing under a write is retried 5 times, then the write throws.
- When more than 1024 delayed children are waiting for their parents and are due earlier than other jobs, those jobs may start up to
FallbackIntervalCheckerlate.
Upgrade
Upgrade all nodes together, and do not roll back to 1.2.3 or earlier once chains exist.
- On Redis, an older version runs only the parent of a new chain, never its children.
- On the other stores, an older version hands out delayed children without waiting for their parent.
From a 1.3.0 prerelease: no further step.
Known limits
- An inline chain child can be left
Idleand not run in these cases: its parent's outcome was stored only by the last write at the stop; its claim failed; the claimed child was put back; or its node was killed while it ran (#84). The log names the child in the first two cases. Nothing claims such a row later; add a new job to run it. - The start of an inline child by its parent's run is not a claim (#83): a child given a time while its parent runs can run twice.
- At-least-once. A job whose process ended before its lost outcome was stored can run again after its rows are released.
- Redis: a node that alone could not reach Redis for about two heartbeat TTLs, while the others could, has its running jobs released (#46).
- Code-defined crons: replicas that start together can still create the same cron twice (#92).
- A child that waits for its parent shows as
Idlein the dashboard.
What's Changed
- Run a due job only on the node that won its claim by @msx752 in #58
- Put back a claim whose job never starts by @msx752 in #59
- Run a cron's Run now on MongoDB and when the claim comes back without its cron by @msx752 in #60
- Start jobs on the task scheduler's own worker threads by @msx752 in #61
- Dispatch the grandchildren of a chain on MongoDB by @msx752 in #62
- Never run a delayed chain child inline, and gate it on the in-memory store (#50, part 1) by @msx752 in #63
- Address the architect review of the delayed-child change (#50, part 1 follow-up) by @msx752 in #64
- Gate delayed chain children on EF Core, and skip only children that have not started (#50, part 2) by @msx752 in #65
- Gate delayed chain children on MongoDB (#50 part 3) by @msx752 in #66
- Start delayed chain children right after their parent ends (#50 part 4) by @msx752 in #67
- Record an inline chain child InProgress while it runs by @msx752 in #68
- Store each Redis chain node as a record of its own (#50 part 5a) by @msx752 in #69
- Gate delayed chain children on Redis (#50 part 5b) by @msx752 in #70
- Release 1.3.0-beta.1 by @msx752 in #71
- Do not start a claimed job whose slot came free as the application stopped by @msx752 in #72
- Retry a put-back that failed inside a claim call by @msx752 in #100
- Add container scenario tests for the fixes on develop by @msx752 in #101
- Bring only the seeded crons to the expression in code (first step of #92) by @msx752 in #102
- Keep a job's end consistent when its outcome cannot be stored by @msx752 in #103
- Fence the writes of a running job by its claim marker by @msx752 in #104
- Write a lost outcome again and fence the stop's Cancelled write by @msx752 in #105
- Add container scenarios for #74 and the first step of #92 by @msx752 in #106
- Hold the worker in the priority tests with a blocking job by @msx752 in #108
- Release what an earlier run left locked when a Redis node starts by @msx752 in #109
- Release a Redis node only after its heartbeat stayed missing for a TTL by @msx752 in #107
- Add a container scenario for the Redis heartbeat after an outage by @msx752 in #110
- Correct five statements of the guide that said more than the code does by @msx752 in #111
- Release 1.3.0-beta.2 by @msx752 in #112
- Give the four stores one rule for claiming a child without a time by id by @msx752 in #113
- Run inline chain children after a parent outcome that was stored late by @msx752 in #114
- Add a container scenario for inline children after a late-stored outcome by @msx752 in #115
- Give the late-loop cron scenario a pause Redis needs since #107 by @msx752 in #116
- Do not start a job before its time when another job is due in the same second by @msx752 in #117
- Make the early-start tests fail for the bug inside the whole suite, and add a container scenario by @msx752 in #118
- Release 1.3.0-beta.3 by @msx752 in #119
Full Changelog: v1.2.3...v1.3.0