Skip to content

Releases

1.6.1

Choose a tag to compare

@msx752 msx752 released this 16 Nov 06:03
de03192

1.6.1 has two fixes, both for Redis as the job store and chains that Redis stored with a version before 1.3.0, which keep their children inside the parent's record until an instance that takes the parent to run it writes them as records of their own: a parent taken by the fallback or run-now while the scheduling loop of the same instance writes those records runs with the children written by then, and a delete of such a chain while those records are written removes them as well. No change of the public API, no new default, no database migration for EF Core, no change of the MongoDB collections, no new Redis key.

Fixed

Redis, chains stored before 1.3.0

  • A parent taken by the fallback or run-now while its children are written as records runs with them (#154). The children of such a chain are written as records in many Redis calls. When the fallback or run-now of an instance read the parent's children while the scheduling loop of the same instance was still writing them, and then took the parent, the parent was run with only the children written before that read; in a test, one of three. A child without a time of its own that was missing was left to the fallback, which takes such a child once its parent ended at least 30 seconds ago. When the record read before the claim held the children inside it, or could not be read, and the claimed record no longer does, the children are now read again after the claim, and the parent runs with the children written by then. If that read fails, a warning is logged, the claim is put back and the parent is not run; the next pass tries again. As in 1.6.0, the scheduling loop does not start the parent a second time: its start is refused once the fallback or run-now holds the record.
  • A delete of such a chain while its children are written as records removes those records (#153). A delete read the chain's children once, so children that the instance writing them linked after that read stayed as Idle records without a parent, which do not run. The delete still removes the children's records first, but keeps their children sets; it then removes the parent's record, reads the chain's children again and removes what it finds, and removes the parent's index entries last. When the last write of the instance that writes the children does not land, or one of its writes fails, that instance checks for the parent's record: when the record is gone, it removes the records it wrote and does not run the parent. If the check or the removal fails, a warning is logged; then, or when that instance stops before it, the records that were not removed stay as Idle records without a parent that do not run.

Behaviour changes to check before upgrading

  • Redis: a delete makes more calls (#153). A delete of a time ticker walks its children a second time, after it removed the ticker's record, and removes the ticker's index entries last; for a chain the number of Redis calls is about twice what it was, for a ticker without children one more. The other stores are not changed.
  • Redis: one more read for a split chain (#154). The fallback and run-now read the children of a chain stored before 1.3.0 again after they took it, when its record no longer holds them. Other jobs are not affected, except one whose record could not be read before the claim.
  • Redis: a delete that stops on a store failure (#153). One that stops before the parent's record is removed leaves the parent listed, and deleting it again removes the rest, as before. One that stops after it leaves no record, only index entries and children sets of records that are gone; reads skip those. Up to 1.6.0 a delete that stopped between its index entries and the parent's record left that record unlisted.

Upgrade

From 1.6.0: no step. Upgrade all instances together, as before.

Known limits

  • Chains stored before 1.3.0 on Redis. When the instance that writes the children of a deleted chain cannot check for the parent's record or remove them, or stops before it, the records it wrote after the delete's last read of the chain's children stay as Idle children without a parent that do not run. A chain whose parent had already ended still keeps its children inside its record, and they are not run; delete it and add it again.
  • Rolling upgrade. The fix of #153 needs both the instance that deletes the chain and the instance that writes its children on 1.6.1; while an instance is still on 1.6.0, records written after the delete read the chain's children can stay as in 1.6.0.
  • Otherwise unchanged from 1.6.0; see its notes.

1.6.0

Choose a tag to compare

@msx752 msx752 released this 15 Nov 09:58
d960ef5

1.6.0 has six fixes: with EF Core, jobs left InProgress under an instance name that does not come back are released, and dotnet ef finds the context of UseSharpTickerDbContext; the dashboard works with ticker types of your own, counts the job statuses in the store and shows when each lock holder last locked a job; and a ticker added before the host starts finds its function. The release of those jobs comes with a new EF Core option, SetInProgressLease(TimeSpan), which is on by default with 5 minutes. No database migration for EF Core, no change of the MongoDB collections, no new Redis key. The option is the one addition to the public API and its default the one new default.

New option

SharpTickerEfCoreOptionBuilder.SetInProgressLease(TimeSpan), on by default with 5 minutes (#24).

options.AddOperationalStore(ef =>
{
    ef.UseApplicationDbContext<AppDbContext>(ConfigurationType.UseModelCustomizer);
    ef.SetInProgressLease(TimeSpan.FromMinutes(5)); // TimeSpan.Zero turns it off
});
  • With the lease on, every instance that runs the background services does this every fifth of the lease (every minute by default), the first time one fifth of the lease after the start: it sets LockedAt to now on the jobs and cron occurrences it holds InProgress since it started (UpdatedAt is not changed), and it releases the jobs and cron occurrences that are InProgress under another instance name and whose LockedAt is older than the lease. A released job is Idle with no lock holder and no LockedAt, as after the startup release, and the fallback runs it again; a chain child waiting for it follows by its run condition.
  • Accepted values: TimeSpan.Zero, which turns the lease off, or a value from 1 second up to 30 days. Any other value throws ArgumentOutOfRangeException.
  • Keep the lease well above the time a renewal pass can take on your database. Set the same lease on every instance that runs the background services, and turn it off on all of them or on none.
  • The lease is part of the EF Core store only. It also runs with Redis next to the store: the heartbeat release frees only the names registered in Redis, the lease also frees names that did not write a heartbeat there; both write the same release. A host with DisableBackgroundServices() neither renews nor releases. A custom provider that does not derive from the EF Core provider of SharpTicker has no lease; the log says so once at Information.
  • The log category is SharpTicker.EntityFrameworkCore.InProgressLease: Information with the number of released jobs and the names that held them (the ten with the most jobs), Warning when a renewal or a release fails; the pass is tried again at the next interval.

See Storage and When an instance stops.

Fixed

EF Core

  • A job left InProgress under an instance name that does not come back is released (#24). With the EF Core store and no Redis, a job that was InProgress when its instance ended without a signal (killed, out of memory, a lost machine) was released only when an instance started again under the same name. A replaced container or a pod of a Deployment gets a new name, so such a job stayed InProgress for good, and a chain child waiting for it waited too. A running instance now releases it once its lease has ended; see "New option".
  • dotnet ef finds the context of UseSharpTickerDbContext (#89). UseSharpTickerDbContext registered an IDbContextFactory<TContext> and a scoped context, but not the non-generic DbContextOptions through which the EF Core tools find a context in the application's services, so in a project with no migrations for it dotnet ef migrations add stopped with "No DbContext was found in assembly ...". That registration is now added, before the other registrations, so an application context that takes the non-generic DbContextOptions still gets its own. Name the project that holds the migrations with MigrationsAssembly, as Storage shows.

Dashboard

  • Lists, add and update work with ticker types of your own (#95). With AddSharpTicker<TTimeTicker, TCronTicker>, the time ticker lists, the occurrence lists of a cron and adding and updating a ticker failed with NotSupportedException on every store, because the dashboard's JSON options knew only the library's types. When reflection-based serialization is on, a type the dashboard's own context does not know is now read and written through reflection; the library's types still go through the context. An edit in the dashboard keeps the stored value of each public property of your own type that the body does not send; the dashboard's dialogs send only the library's fields, and on EF Core an update writes every column, so without this an edit would empty them. A body that sends such a property still sets it.
  • The status counts are counted by the store (#96). The time ticker chart (/time-tickers/graph-data), the cron chart (/cron-tickers/graph-data) and the overall statistics (/ticker/statuses/get) read every visible row, on EF Core with its children or its cron, and grouped them in memory. They now count each status through TimeTickersQuery() and CronTickerOccurrencesQuery(): on EF Core and MongoDB the database counts. The in-memory and Redis stores read the rows once and count them in memory, as before. The jobs per machine (/ticker/machine/jobs) still read the locked rows, without their children or cron. The counts are made of the same rows as before: top-level time tickers, the visibility rule applied, the occurrences of the visible crons.
  • The Lock holders panel shows when each name last locked a job and, with Redis, whether it has a heartbeat (#31). The Machines panel listed every node identifier that had locked a job still in the store, ordered by the number of jobs, so the names of stopped or replaced containers with many old rows came first and looked like running machines. The panel is now called Lock holders. Each name shows its number of jobs and how long ago it last locked one, latest first; with Redis also heartbeat or no heartbeat. The EF Core, MongoDB and in-memory stores keep no record of which nodes run, so there the panel shows the names and their latest lock only. No name is hidden.

Startup

  • A ticker added after UseSharpTicker and before the host starts finds its function (#75). The function registry was built only when the host started, so a time or cron ticker added between app.Build() and app.Run(), or by a hosted service that starts before SharpTicker's, failed with "Cannot find TickerFunction with name ...". UseSharpTicker now builds the registry; the build has no I/O. The start builds it again, so functions mapped after UseSharpTicker are found from the start on, and a cron from the configuration that is not a valid expression still stops the start.

Behaviour changes to check before upgrading

  • EF Core: old InProgress jobs run again (#24). At the first pass after the upgrade, about a minute after the start with the default lease, every job and cron occurrence that is InProgress under another instance name with a LockedAt older than the lease is released and runs again, however old it is: jobs left by replaced containers or pods can be weeks old, and cron occurrences run late. A chain child waiting for such a job follows by its run condition. Look for these jobs before the upgrade: the startup log of 1.5.0 and later names the instance names that hold them (Information, category SharpTicker.EntityFrameworkCore.StartupRelease); in the database they are the rows with Status 2 (InProgress) in TimeTickers and CronTickerOccurrences, with their LockHolder and LockedAt. Delete the ones that should not run (Force delete on the dashboard for a time job), or upgrade with SetInProgressLease(TimeSpan.Zero) and turn the lease on once they are dealt with. A row with no LockedAt or no lock holder is not released by the lease.
  • EF Core: the same lease on every instance (#24). An instance of an earlier version, or one with the lease off, does not renew its locks, so an instance with the lease on releases the jobs that instance runs for longer than the lease, and they run twice. With different values, the shorter lease releases the jobs of the others earlier than their lease expects. Upgrade all instances together.
  • EF Core: "Locked At" shows the last renewal (#24). For a job that is InProgress, LockedAt and the dashboard's "Locked At" show when the lock was last renewed, not when the job was claimed; so does the latest lock of a name in the Lock holders panel.
  • EF Core: dotnet ef needs --context in a project with a context of its own (#89). In a project that registers a context of its own (AddDbContext<AppDbContext>) and also UseSharpTickerDbContext, with no migrations for SharpTickerDbContext, the tools picked the application's context up to 1.5.2 when no --context was given; now they find both and stop with "More than one DbContext was found". Scripts that run dotnet ef database update, dotnet ef migrations script or dotnet ef migrations bundle in such a project need --context AppDbContext. The options action given to UseSharpTickerDbContext also runs once more, when the new registration is resolved.
  • Dashboard: the public properties of your own ticker types are shown and accepted (#95). The public properties that your TTimeTicker and TCronTicker add are in the dashboard's answers and are read from the bodies of add and update. Mark a property [JsonIgnore] to keep it out of both; such a property keeps its stored value on an edit.
  • **Dashboard: the request ch...
Read more

1.5.2

Choose a tag to compare

@msx752 msx752 released this 15 Nov 03:47
75233cf

1.5.2 fixes five faults reported against 1.5.1 and earlier: a cron expression that stopped the scheduler loop, numbers of an application's own ticker type that Redis rewrote, chain children whose time was read in the wrong zone or whose interrupted run was skipped, and a stored cron ticker that did not run without a word. No database migration for EF Core, no change of the MongoDB collections, no new Redis key, no change of the public API or of a default.

Fixed

Scheduling

  • A cron whose expression matches no date no longer stops the scheduler loop (#194). An expression such as 0 0 0 30 2 * (the 30th of February) parses, and WithCron in code accepts it. The cron library threw for it, so while such a cron ticker was stored the loop failed on every pass and started no job on time, cron or time; only the fallback started overdue jobs, and the dashboard's list of cron jobs failed. The loop now skips that cron ticker, and adding or updating one is refused. A cron seeded from code with such an expression is still stored; the startup Warning of #79 names it.
  • A stored cron ticker that does not run is named in a Warning at startup (#79). Each enabled cron ticker whose expression has no next occurrence (five fields stored by a version before 1.5.0, a configuration placeholder whose key is missing, an expression that matches no date) is named once with its id, its function and the expression, on each instance. The check does not stop the startup.

Chains

  • The times of delayed children and grandchildren are converted to UTC like the root's (#81). When a chain was added, a child time given as DateTimeKind.Local or Unspecified was stored as given, so on a machine not on UTC the child ran hours off. When a chain is added through the dashboard API, a timeZoneId now applies to every time of the chain.
  • A chain child that starts with its parent and was left waiting runs when its parent started (#158). A child with RunCondition.InProgress and no time of its own that still waited after its parent ended was recorded Skipped, also when its run had been interrupted by a stopped node and released. The fallback now runs it when the parent started (Done, DueDone, Failed, Cancelled) and skips it when the parent was skipped.

Redis as the job store

  • Numbers of an application's own ticker type are kept (#151). Claims, releases and skips were Lua scripts that rewrote a record with the JSON library of Redis, which keeps 14 significant digits: a long of 15 digits or more came back in exponent form, the record could not be read and the job did not run; a long double was rounded. The store now makes these writes with the serializer that wrote the record.

Behaviour changes to check before upgrading

  • Delayed chain children with Local or Unspecified times now run at the time the caller meant. A caller that built an Unspecified value meaning UTC on a machine not on UTC gets it shifted by the scheduler zone's offset, as the root's was. Times already stored are not changed.
  • InProgress chain children left waiting run (DueDone, 30 seconds plus up to one FallbackIntervalChecker after the parent's end) where they were Skipped. A child killed with its node can run twice, as any released job can. With the Redis heartbeat's self-fence, which does not ask such a child to stop, a child still running on a cut-off node can run a second time at the same time once its row is released. A child 1.5.1 already recorded as Skipped stays so.
  • Redis: upgrade all instances together. An instance still on 1.5.1 rewrites a record when it claims it. A record 1.5.1 already wrote back in exponent form is not repaired, because its digits are lost; it is named in a warning with its key: delete or update that job.
  • Redis: a record that cannot be read is no longer taken by the fallback and left InProgress; it stays as it is and is named on each read.
  • Redis: each claim, release and skip is a read and a compare-and-set, two round trips where a script was one.
  • A cron expression that matches no date is refused when a cron ticker is added or updated, with "Cannot parse expression ..., or it matches no date".

Upgrade

From 1.5.1: no schema or data step. With Redis, upgrade all instances together. Correct or remove the cron tickers the startup warning names.

Known limits

Unchanged from 1.5.1; see its notes.

1.5.1

Choose a tag to compare

@msx752 msx752 released this 14 Nov 16:21
87dd598

1.5.1 fixes six faults reported against 1.5.0 and earlier: jobs that started late through the fallback, a second run of chain children stored by a version before 1.3.0, and claims that could be left InProgress or tried again for nothing. No database migration for EF Core, no change of the MongoDB collections, no new Redis key, no change of the public API or of a default.

Fixed

Jobs that started late

  • A job added while the scheduler loop reads its next plan starts at its time (#78). The loop did not see the new job in that read and was not woken for it; the fallback started it, up to FallbackIntervalChecker (30 seconds by default) late, and it ended DueDone.
  • EF Core or MongoDB with the Redis heartbeat: the release of a stopped node writes only that node's jobs (#90). It also wrote every waiting job no node held, which made the scheduler loop of the live nodes miss them; they started late through the fallback.

Redis as the job store

  • An edit of a chain stored before 1.3.0 whose parent already ended runs no child (#152). Saving such a chain (UpdateAsync, or the dashboard's edit) wrote its children as records of their own, and the fallback ran the ones the parent's outcome lets run, although 1.3.x had already run them. The edit now changes the parent's record only.
  • No pending entry is left on a job that ended (#57). A put-back that met a job ending at the same moment left one for good, and the fallback of cron occurrences never dropped the entry of an occurrence that had ended; each cost one claim attempt per fallback pass.

Claims

  • A claimed job is put back later also when the store failed twice in a row during its claim (#155): the put-back of the row being claimed failed, then the put-back of the rows already won. It stayed InProgress for its node until the node restarted or its jobs were released.
  • A job whose slot came free as the application stopped does not start (#88). It would have run with a cancelled token and ended Cancelled. No caller in the library reaches this path today.

Behaviour changes to check before upgrading

  • Redis, chains stored before 1.3.0 whose parent already ended: an edit no longer runs their children, and what the edit carries for those children is not stored. To run them, delete the chain and add it again.
  • Fewer jobs end DueDone: jobs that started late through the fallback in the cases above start at their time.

Upgrade

From 1.5.0: no step. As with every release, an instance still on 1.5.0 behaves as 1.5.0 did: it still writes the children of such a chain when it is edited.

Known limits

Unchanged from 1.5.0; see its notes. The in-memory store's release of a stopped node still writes waiting jobs no node holds (#90 covers EF Core and MongoDB; nothing calls that release of the in-memory store in a running application).

1.5.0

Choose a tag to compare

@msx752 msx752 released this 14 Nov 11:31
fecc906

1.5.0 is mostly about Redis as the job store: a node that loses its connection to Redis now stops its own jobs before another node starts them again, applications with ticker types of their own can use the store, and records left by a version before 1.3.0 are read. It also fixes crons that never ran (a five-field value from configuration, the official MySQL provider on net8.0) and a cron that kept its old schedule after a deploy. No database migration for EF Core, no change of the MongoDB collections, no new Redis key. Please read Behaviour changes before deploying: with Redis as the job store, running jobs can now be asked to stop and run again in a Redis outage.

Fixed

Redis as the job store

  • A node cut off from Redis stops its jobs before they are started again (#46). A node that cannot write its heartbeat for one heartbeat TTL (80 seconds with the default interval) asks its running jobs to stop and starts none until its heartbeat is written again. Jobs that stop are put back and run again. Before, the other nodes released the jobs of the silent node while they could still be running on it. StopJobsWhenHeartbeatIsLost = false on the Redis options turns this off.
  • An application's own ticker types work (#162). Adding a job failed without a JsonSerializerContext, and with a source-generated one the scheduler loop and the fallback failed on reading the jobs. Both work now, without a setting for the serializer.
  • Records left by a version before 1.3.0 are read (#144, #145). Time jobs that such a version left InProgress are released when the instance starts under the NodeIdentifier that holds them, and run. A chain stored inside its parent's record gets its children written as records of their own when the parent is taken to run, and they run by their run conditions.
  • A job whose own ticker type has an empty collection runs (#146). Such a job was claimed and not handed out, and stayed InProgress.
  • One record that cannot be read no longer stops the others (#150). The record is left out of the read and logged as a warning with its key.
  • A job added shortly before it is due starts at its time (#176). The scheduler loop did not take such a job when the last digit of its UpdatedAt was a zero, about one job in ten; the fallback started it up to FallbackIntervalChecker (30 seconds by default) late.

Crons

  • A cron from configuration (%Key%) with five fields runs (first part of #79). It was stored as it was and never ran. A value that is not a cron expression now stops the startup with an error that names the function, the key and the value.
  • Crons run on the official MySQL provider on net8.0 (#91). SharpTicker.EntityFrameworkCore asks for FlexLabs.EntityFrameworkCore.Upsert 8.1.2 or later on net8.0.
  • A cron changed in code runs on its new schedule after a deploy, with the EF Core store and Redis next to it (#93). The list of cron expressions kept in Redis is cleared when the crons are seeded at startup and ends ten minutes after it was read.

Other

  • A job whose cancellation callback throws no longer breaks the stop of the application (#73). The stop logs a warning and waits for the running jobs as before.
  • MongoDB: the fallback takes a job only as it read it (#55). A job rescheduled in between could run early with its old payload.

New in the log

  • EF Core without Redis: a starting instance writes one Information entry that names the other node identifiers holding InProgress jobs, with their counts (first part of #24). Nothing is released by it.
  • Redis as the job store: an Error when the node stops its jobs because its heartbeat could not be written, and an Information entry when it starts jobs again.
  • EF Core with Redis: a Warning when the cached cron list cannot be cleared at startup.

Behaviour changes to check before upgrading

  • With Redis as the job store, running jobs are asked to stop when the node cannot write its heartbeat for one TTL, and they run again later. This also happens in a Redis outage that every instance sees, where the jobs ran on before. Check that your jobs honour their cancellation token and are safe to run twice. Jobs that must not be interrupted that way need StopJobsWhenHeartbeatIsLost = false.
  • Stored jobs that did not run before can run after the upgrade. On a Redis store that was ever used by a version before 1.3.0: time jobs left InProgress, and the children of chains whose parent has not run yet. With ticker types of your own: jobs that stayed InProgress because of an empty collection. Delete the ones that should not run before upgrading; Upgrading says where to look.
  • A configured cron value that is not a cron expression stops the startup. Up to 1.4.0 the application started and the job never ran. A five-field value that never ran starts to run. Check the configured values.
  • EF Core on net8.0: a project that pins FlexLabs.EntityFrameworkCore.Upsert below 8.1.2 gets a restore error until the pin is raised.
  • EF Core with Redis: a cron list entry written by 1.4.0 or earlier has no end. The first start on 1.5.0 clears it when it seeds; an instance still on the earlier version writes it again.
  • Redis: jobs added shortly before they are due start earlier than they did, and fewer end DueDone.

Upgrade

From 1.4.0: no schema step. Upgrade all instances together: an instance on 1.4.0 behaves as 1.4.0 did for the jobs it runs, takes a chain stored before 1.3.0 without its children, and writes the cron list entry without an end.

Known limits

  • A job that ignores its cancellation token runs on after its node was cut off from Redis, and can then run a second time on another node.
  • The stop of a node's jobs applies to Redis as the job store. With the EF Core or MongoDB store and the Redis heartbeat next to it, nothing changes.
  • EF Core without Redis: jobs held under a node name that no instance starts with again stay InProgress (#24).
  • Redis with ticker types of your own: a number with more than 14 digits in such a type is rewritten by the store's scripts, and the job is not handed out (#151). Values of abstract types need polymorphism configured.
  • A chain stored before 1.3.0 whose parent already ended is left as it is; delete it and add it again for its children to run.
  • A stored cron that has no next occurrence logs nothing (#79).
  • MongoDB has no release step for a node that stopped (#140).

1.4.0

Choose a tag to compare

@msx752 msx752 released this 12 Nov 17:31
2aeab18

1.4.0 closes the two known gaps in how a chain child without a time of its own is started: it could run twice after an edit, and it could be left Idle for good. No database migration for EF Core, no change of the MongoDB collections. Redis gets one new key. Please read Behaviour changes before deploying: children that an earlier version left Idle are run or skipped after the upgrade.

Fixed

Chain children without a time of their own

  • A child given a time while its parent runs no longer runs twice (#83). The parent's run now starts such a child with a claim on its stored row, and the row refuses the claim once it has a time, is held, or is no longer Idle. Before, the parent's run started the child whatever its row said, while the store handed the same child out at its new time.
  • A child whose start claim fails is not run without its row (#83). Before, a failed write at the child's start was logged and the child ran anyway, with nothing fencing its writes. It now keeps waiting and a Warning names it.
  • A child that was never started is run by the fallback (#84). A child that is Idle, has no time and is not held, and whose parent ended at least 30 seconds ago, is taken by the fallback: it runs as a job of its own and ends DueDone when the parent's outcome lets it run, and is recorded as Skipped otherwise. This covers a node that was killed while it ran the child (EF Core and Redis, after the node's rows are released), a node that stopped between the parent's end and the child's start, and a start claim that failed. Before, nothing took such a row.

Behaviour changes to check before upgrading

  • Children left Idle by earlier versions are decided after the upgrade. Every chain child without a time whose parent has ended is run or skipped by the first fallback pass, however old it is. Delete the ones you do not want to run before upgrading.
  • Such a child starts later than before in the failure cases: 30 seconds plus up to one FallbackIntervalChecker (30 seconds by default) after its parent's end, as a job of its own with its own concurrency slot and priority. It ends DueDone, not Done.
  • A child whose parent's outcome does not let it run becomes Skipped where it stayed Idle.
  • Every fallback pass does one more read. On EF Core it is one query without a new index. On MongoDB and Redis its cost grows with the number of chains whose parent has not run yet: about one second per pass with 100 000 such chains, measured on one machine.
  • Redis: one new key, the hash tq:tt:waiting-children. The first fallback pass of each process fills it from the stored jobs; with very many stored time tickers that first pass takes longer.
  • Log texts changed: the entries that told you to add a new job for a child that did not start now say that the fallback takes it.

Upgrade

From 1.3.1: no step. Upgrade all nodes together where you can. While a 1.3.x node still runs, it starts chain children the old way, so the two fixes above do not hold for the chains it runs, and on Redis a child it stores gets no entry in the new key until a 1.4.0 process starts.

Known limits

  • MongoDB has no release step for a node that stopped: a job, and so also a chain child, that was InProgress when its node was killed stays InProgress (#140).
  • At-least-once: a job whose process ended before its lost outcome was stored can run again.
  • Redis: a node that alone could not reach Redis for about two heartbeat TTLs has its running jobs released (#46).
  • A child skipped by the fallback is followed by what it carries one level per pass: each level is skipped 30 seconds and one pass after the level above it.

What's Changed

  • Start an inline chain child only while its row still waits by @msx752 in #133
  • Do not run an inline chain child whose start claim was not won by @msx752 in #135
  • Let the fallback take a chain child without a time that was never started by @msx752 in #136
  • Docs: the start of a chain child without a time, and the fallback that takes one left waiting by @msx752 in #137
  • Add container scenarios for the start of a chain child without a time by @msx752 in #138
  • Release 1.4.0 by @msx752 in #141

Full Changelog: v1.3.1...v1.4.0

1.3.1

Choose a tag to compare

@msx752 msx752 released this 12 Nov 10:35
d556dee

1.3.1 fixes the last known way a code-defined cron could run twice, two faults in editing chain children, and three smaller ones. No database migration for EF Core, no change of the MongoDB collections, no new Redis keys. Please read Behaviour changes before deploying.

Fixed

Code-defined crons (#92)

  • Replicas that start together no longer create the same cron twice. The row the seed inserts gets an id made from the function name, the same on every node, on EF Core, MongoDB and Redis. Rows that exist keep their ids.
  • Copies made by earlier versions are removed at startup. Of the rows that carry the seed's marker for a function, the oldest is kept; the others are deleted with their occurrences, and a Warning names the function and the number removed. A cron added through the API or the dashboard, and a seeded cron edited on the dashboard, are not touched.

Chain children

  • Editing a child no longer takes it out of its chain (#85). UpdateAsync and the dashboard edit keep the parent of a stored child, and its run condition when the update carries none. Before, on EF Core, MongoDB and the in-memory store the child became a job of its own and ran at its time whatever its parent did.
  • A child without a time of its own can be edited and stay that way (#85). The dashboard dialog no longer asks for a time for such a child, and shows its run condition.
  • A child whose function is not registered gets a final status (#82). It ends Failed with a message that names the function, and its own children follow that outcome. Before, it was left Idle and nothing reported it.

Scheduler

  • A cancelled call no longer queues its job into a full queue (#86). QueueAsync with a token cancelled while it waited for room could return without an exception and add the job anyway.
  • The restart timer does nothing after its manager is disposed (#77). A callback already queued at the stop could still ask for a restart.

Behaviour changes to check before upgrading

  • Stored copies of a seeded cron are deleted at the first start, with their occurrence history. Only rows with InitIdentifier = "MemoryTicker_Seeded_" + function count, and one of them always stays.
  • A chain child without a registered function now ends Failed where it stayed Idle. Filters and alerts on Failed see these rows.
  • UpdateAsync keeps ParentId of a stored child even when the update names another parent or none. To move a child, delete it and add a new job.
  • UpdateAsync without an ExecutionTime now succeeds for a stored child that has no time. For every other job it is refused as before.
  • Dashboard: "Execute immediately" is no longer offered when a stored job is edited (the update refused it); it is still offered for a new job and a copy.

Upgrade

From 1.3.0: no step. Nodes of 1.3.0 and 1.3.1 can run together; a 1.3.0 node that starts together with another node can still insert a copy of a seeded cron, and the next start of a 1.3.1 node removes it.

Known limits

Unchanged from 1.3.0, except the entry on code-defined crons, which is fixed:

  • An inline chain child can be left Idle and not run in the cases listed for 1.3.0 (#84).
  • The start of an inline child by its parent's run is not a claim (#83).
  • At-least-once: a job whose process ended before its lost outcome was stored can run again.
  • Redis: a node that alone could not reach Redis for about two heartbeat TTLs has its running jobs released (#46).
  • A seeded cron whose function was changed keeps its id. If a new function gets that name, its cron is seeded under a random id, and that one start has no guard against a second node.

What's Changed

  • Start the queued work on the calling thread in the hand-over tests by @msx752 in #120
  • Do nothing in the restart timer after its manager is disposed by @msx752 in #121
  • Give a chain child without a registered function a final status by @msx752 in #122
  • Keep a chain child in its chain when it is edited by @msx752 in #123
  • EF Core: seed a code-defined cron once when nodes start together by @msx752 in #124
  • Do not queue a cancelled caller's job into a full queue by @msx752 in #125
  • MongoDB: seed a code-defined cron once when nodes start together by @msx752 in #126
  • Redis: seed a code-defined cron once when nodes start together by @msx752 in #127
  • EF Core: remove the stored copies of a seeded cron at startup by @msx752 in #128
  • MongoDB, Redis: remove the stored copies of a seeded cron at startup by @msx752 in #129
  • Let a chain child without a time be edited without getting one by @msx752 in #130
  • Add a container scenario for nodes that seed a cron together by @msx752 in #131
  • Release 1.3.1 by @msx752 in #132

Full Changelog: v1.3.0...v1.3.1

1.3.0

Choose a tag to compare

@msx752 msx752 released this 11 Nov 21:47
54ea55f

1.3.0 changes how chains run on every store and fixes several ways a job could run twice, never run, start early or stay InProgress. It is the code of 1.3.0-beta.3. Please read Upgrade before deploying. No database migration for EF Core and no change of the MongoDB collections; Redis gets new keys (see Redis).

Delayed chain children (#50)

A child or grandchild with an execution time of its own (SetExecutionTime) now waits for its parent on every store: EF Core, MongoDB, Redis and in-memory.

  • It runs only after its parent ended with a status its RunCondition accepts, and not before its own time.
  • If that time has already passed, it starts right after the parent and ends Done. One started by the fallback ends DueDone.
  • When the parent ends with a status the condition does not accept, the child is recorded Skipped, with everything below it.

Before, a delayed child ran right after its parent and its own time was ignored; on EF Core and MongoDB it was handed out at its own time, without waiting for its parent.

Fixed

Jobs that ran twice

  • A due job runs once, on the node that won its claim (#45). The scheduler loop and the fallback could both run the same row, on several nodes or on one.
  • A late write of an old run no longer overwrites a newer run (#74). Every write of a running job is fenced by the marker its claim wrote.
  • Saving a whole time ticker no longer runs it again (#56). UpdateAsync and the dashboard edit keep the run state of a row that is running or has ended.
  • Redis: a live node's running jobs are no longer released after a Redis outage (#107). A node whose heartbeat is missing is released only when it stayed missing for one more heartbeat TTL (NodeHeartbeatInterval + 20 s), and a node never releases itself.

Jobs that never ran or stayed InProgress

  • A job's outcome is no longer lost when its final write fails (#74, #80). The scheduler writes the outcome again while the instance runs, at most 5 seconds apart, until the store takes it; while outcomes wait it logs an Error from 5 minutes and a Critical from 10. A stop no longer waits for such a job and its cron's next occurrence is no longer skipped.
  • Chain children follow a parent outcome that was stored late (#74). Those it lets run are claimed from the store and run, those it does not let run are skipped with their descendants.
  • A claim whose job never starts is put back (#44, #72, #100). This covers an application that stops while the job waits for a slot, a frozen or refusing scheduler, and a claim call that failed after its write landed.
  • Redis: a node that starts again under the same name releases what its earlier run left locked (#109).
  • MongoDB: "Run now" on a cron works (#52), and grandchildren of a chain are dispatched (#51).

Other

  • A job no longer starts before its time when another job is due in the same second (#117). It could start up to one second early.
  • Jobs of functions without a request type start on the scheduler's own worker threads (#61), so a busy thread pool no longer delays them. A function with a typed request still starts its code on the .NET thread pool.
  • Code-defined crons (first step of #92). Redis no longer fails to start when a function has two crons. EF Core and MongoDB no longer overwrite a user's cron of the same function: only the seeded rows take the expression in code.

Behaviour changes to check before upgrading

  • Rows stored before the upgrade follow the new chain rules on EF Core and MongoDB. A pending delayed child whose parent has not ended waits. One whose parent already ended without a match is recorded Skipped.
  • An inline child (one without a time of its own) is recorded InProgress while it runs (#68). This costs one store write per inline child before it starts.
  • A child that follows a late-stored outcome runs as a job of its own: it takes its own concurrency slot and priority.
  • UpdatedAt of a running job holds the claim marker for the whole run; it no longer moves with every write of the run.
  • An ended job is not run again by saving it. Add a new job instead.
  • Writes during a run (a failed attempt, the next retry number) are best effort: a failure is logged as a Warning and no longer fails the job.
  • Several crons of one function, none marked as seeded: the seed changes none of them and logs a Warning at every start.
  • Filters on DueDone see fewer rows.
  • Custom persistence providers are not fenced and do not follow late-stored outcomes; for those points they keep the behaviour of 1.2.3.

Redis

  • Each chain child is a record of its own, and the dashboard shows its real status (#69). New keys: tq:tt:{id} per child, plus tq:tt:{id}:children.
  • Chains stored by 1.2.3 or earlier keep their children nested in the parent's record and keep the old behaviour: a delayed child runs right after its parent. The dashboard does not show those children's results, and such a child cannot be deleted on its own; delete the chain.
  • A dead node is released later: about two heartbeat TTLs after its last heartbeat, not one.
  • One name per running instance. Two processes that run at once under one NodeIdentifier release each other's running jobs at startup, as with EF Core.
  • Writes are bounded. A row that keeps changing under a write is retried 5 times, then the write throws.
  • When more than 1024 delayed children are waiting for their parents and are due earlier than other jobs, those jobs may start up to FallbackIntervalChecker late.

Upgrade

Upgrade all nodes together, and do not roll back to 1.2.3 or earlier once chains exist.

  • On Redis, an older version runs only the parent of a new chain, never its children.
  • On the other stores, an older version hands out delayed children without waiting for their parent.

From a 1.3.0 prerelease: no further step.

Known limits

  • An inline chain child can be left Idle and not run in these cases: its parent's outcome was stored only by the last write at the stop; its claim failed; the claimed child was put back; or its node was killed while it ran (#84). The log names the child in the first two cases. Nothing claims such a row later; add a new job to run it.
  • The start of an inline child by its parent's run is not a claim (#83): a child given a time while its parent runs can run twice.
  • At-least-once. A job whose process ended before its lost outcome was stored can run again after its rows are released.
  • Redis: a node that alone could not reach Redis for about two heartbeat TTLs, while the others could, has its running jobs released (#46).
  • Code-defined crons: replicas that start together can still create the same cron twice (#92).
  • A child that waits for its parent shows as Idle in the dashboard.

What's Changed

  • Run a due job only on the node that won its claim by @msx752 in #58
  • Put back a claim whose job never starts by @msx752 in #59
  • Run a cron's Run now on MongoDB and when the claim comes back without its cron by @msx752 in #60
  • Start jobs on the task scheduler's own worker threads by @msx752 in #61
  • Dispatch the grandchildren of a chain on MongoDB by @msx752 in #62
  • Never run a delayed chain child inline, and gate it on the in-memory store (#50, part 1) by @msx752 in #63
  • Address the architect review of the delayed-child change (#50, part 1 follow-up) by @msx752 in #64
  • Gate delayed chain children on EF Core, and skip only children that have not started (#50, part 2) by @msx752 in #65
  • Gate delayed chain children on MongoDB (#50 part 3) by @msx752 in #66
  • Start delayed chain children right after their parent ends (#50 part 4) by @msx752 in #67
  • Record an inline chain child InProgress while it runs by @msx752 in #68
  • Store each Redis chain node as a record of its own (#50 part 5a) by @msx752 in #69
  • Gate delayed chain children on Redis (#50 part 5b) by @msx752 in #70
  • Release 1.3.0-beta.1 by @msx752 in #71
  • Do not start a claimed job whose slot came free as the application stopped by @msx752 in #72
  • Retry a put-back that failed inside a claim call by @msx752 in #100
  • Add container scenario tests for the fixes on develop by @msx752 in #101
  • Bring only the seeded crons to the expression in code (first step of #92) by @msx752 in #102
  • Keep a job's end consistent when its outcome cannot be stored by @msx752 in #103
  • Fence the writes of a running job by its claim marker by @msx752 in #104
  • Write a lost outcome again and fence the stop's Cancelled write by @msx752 in #105
  • Add container scenarios for #74 and the first step of #92 by @msx752 in #106
  • Hold the worker in the priority tests with a blocking job by @msx752 in #108
  • Release what an earlier run left locked when a Redis node starts by @msx752 in #109
  • Release a Redis node only after its heartbeat stayed missing for a TTL by @msx752 in https://github.com/msx752/Shar...
Read more

v1.3.0-beta.3

v1.3.0-beta.3 Pre-release
Pre-release

Choose a tag to compare

@msx752 msx752 released this 11 Nov 17:59
54ea55f

A prerelease. It closes the known limit of 1.3.0-beta.2 for chains: the children without a time of their own now follow a parent outcome that was stored late. No public API change and no database schema change since 1.3.0-beta.2.

Fixed

  • Inline chain children run after a parent outcome that was stored late (#74). When a parent's final write failed and the scheduler stored it later, the parent's children without a time of their own stayed Idle, neither run nor skipped. They now follow the stored outcome: those it lets run are claimed from the store and run, those it does not let run are skipped with their descendants.
  • One rule on every store for claiming a chain child without a time by id (#99). Such a child is handed out by id only once its parent has ended with an outcome it runs on, and an InProgress child without a time never is. Redis refused every such claim before; EF Core, MongoDB and the in-memory store also allowed it for InProgress children. Nothing used this claim before this prerelease.
  • A job no longer starts before its time when another job is due in the same second (#117). The scheduler plans the jobs of one second together and started them all at the earliest of their times, so a time ticker or a delayed child could start up to one second early beside another time ticker or a cron occurrence. Each now starts at its own time. 1.3.0-beta.2 listed this as a known limit for cron occurrences only.

Behaviour changes to check before upgrading

  • A child that follows a late-stored outcome runs as a job of its own, not inside its parent's run: it takes its own concurrency slot and priority, and it ends Done also when the fallback had started its parent (which then ended DueDone). A child whose parent's outcome was stored at once runs inside the parent's run, as before.
  • Custom persistence providers are not fenced, so a lost outcome is not written again there and none of this applies.

Known limits in this prerelease

  • An inline chain child can still be left Idle and not run in these cases: its parent's outcome was stored only by the last write at the stop; its claim failed; the claimed child was put back; or its node was killed while it ran (#84). The log names the child in the first two cases. Nothing claims such a row later; add a new job to run it.
  • The start of an inline child by its parent's run is not a claim (#83): a child given a time while its parent runs can run twice.
  • At-least-once. A job whose process ended before its lost outcome was stored can run again after its rows are released.
  • Redis: a node that alone could not reach Redis for about two heartbeat TTLs, while the others could, has its running jobs released (#46).
  • Code-defined crons: replicas that start together can still create the same cron twice (#92).

What's Changed

  • Give the four stores one rule for claiming a child without a time by id by @msx752 in #113
  • Run inline chain children after a parent outcome that was stored late by @msx752 in #114
  • Add a container scenario for inline children after a late-stored outcome by @msx752 in #115
  • Give the late-loop cron scenario a pause Redis needs since #107 by @msx752 in #116
  • Do not start a job before its time when another job is due in the same second by @msx752 in #117
  • Make the early-start tests fail for the bug inside the whole suite, and add a container scenario by @msx752 in #118
  • Release 1.3.0-beta.3 by @msx752 in #119

Full Changelog: v1.3.0-beta.2...v1.3.0-beta.3

v1.3.0-beta.2

v1.3.0-beta.2 Pre-release
Pre-release

Choose a tag to compare

@msx752 msx752 released this 11 Nov 12:34
88f8d78

A prerelease. It makes the end of a job reliable when the store fails at the wrong moment, and stops the Redis heartbeat from starting running jobs a second time. No public API change and no database schema change since 1.3.0-beta.1.

Fixed

  • A job's outcome is no longer lost when its final write fails (#74, #80). The job leaves the list of running jobs on every path, so a stop no longer waits for it and its cron's next occurrence is no longer skipped. The scheduler writes the outcome again while the instance runs, at most 5 seconds apart, until the store takes it; while outcomes wait it logs an Error from 5 minutes and a Critical from 10. Chain children are run, handed off or skipped only once the parent's outcome is stored.
  • A late write of an old run no longer overwrites a newer run (#74). Every write of a running job is fenced by the marker its claim wrote. The stop's Cancelled write is fenced too and no longer replaces a stored outcome.
  • Saving a whole time ticker no longer runs it again (#56). UpdateAsync and the dashboard edit keep the run state of a row that is running or has ended.
  • Redis: a live node's running jobs are no longer released after a Redis outage (#107). A node whose heartbeat is missing is released only when it stayed missing for one more heartbeat TTL (NodeHeartbeatInterval + 20 s), and a node never releases itself. Before, after an outage longer than the TTL, the first node back released the others' running jobs and they started a second time.
  • Redis: a node that starts again under the same name releases what its earlier run left locked (#109). Before, those jobs stayed InProgress and never ran again; a single Redis instance had no release at all.
  • A put-back that failed inside a claim call is retried (#100). A job could stay InProgress with no worker until the node restarted.
  • Code-defined crons (first step of #92). Redis no longer fails to start when a function has two crons. EF Core and MongoDB no longer overwrite a user's cron of the same function: only the seeded rows take the expression in code.
  • A job waiting for a concurrency slot no longer starts while the application stops (#72); it goes back to Idle.

Behaviour changes to check before upgrading

  • UpdatedAt of a running job holds the claim marker for the whole run; it no longer moves with every write of the run.
  • An ended job is not run again by saving it. Add a new job instead.
  • Redis releases a dead node later: about two heartbeat TTLs after its last heartbeat, not one.
  • Redis, one name per running instance. Two processes that run at once under one NodeIdentifier now release each other's running jobs at startup, as with EF Core. Give each instance its own name.
  • Redis writes are bounded. A row that keeps changing under a write is retried 5 times, then the write throws.
  • Writes during a run (a failed attempt, the next retry number) are best effort: a failure is logged as a Warning and no longer fails the job.
  • Several crons of one function, none marked as seeded: the seed changes none of them and logs a Warning at every start.
  • Custom persistence providers are not fenced: they keep the behaviour of 1.3.0-beta.1 for all of the above.

Known limits in this prerelease

  • Inline chain children after an outcome that was stored late. When a parent's final write fails and the scheduler stores it later, the parent's children without a time of their own are neither run nor skipped; they stay Idle. The log names the parent. Delayed children are not affected. This is the remaining step of #74 and will be fixed before 1.3.0.
  • An inline chain child whose node was killed is released to Idle and not run again (#84).
  • At-least-once. A job whose process ended before its lost outcome was stored can run again after its rows are released.
  • Redis: a node that alone could not reach Redis for about two heartbeat TTLs, while the others could, has its running jobs released (#46).
  • Code-defined crons: replicas that start together can still create the same cron twice (#92).
  • A job can start up to one second before its time when a cron occurrence is due in the same second.

Corrections to the 1.3.0-beta.1 notes

  • "Jobs start on the scheduler's own worker threads" (#61) holds for functions without a request type. A function with a typed request still starts its code on the .NET thread pool.
  • The put-back of a claim whose job never starts (#44, #54) was complete only for a put-back that succeeds at the first try; the retry of a failed put-back comes with this prerelease (#100).
  • A delayed chain child "never starts before its own time": it can start up to one second early (see above).

What's Changed

  • Do not start a claimed job whose slot came free as the application stopped by @msx752 in #72
  • Retry a put-back that failed inside a claim call by @msx752 in #100
  • Add container scenario tests for the fixes on develop by @msx752 in #101
  • Bring only the seeded crons to the expression in code (first step of #92) by @msx752 in #102
  • Keep a job's end consistent when its outcome cannot be stored by @msx752 in #103
  • Fence the writes of a running job by its claim marker by @msx752 in #104
  • Write a lost outcome again and fence the stop's Cancelled write by @msx752 in #105
  • Add container scenarios for #74 and the first step of #92 by @msx752 in #106
  • Hold the worker in the priority tests with a blocking job by @msx752 in #108
  • Release what an earlier run left locked when a Redis node starts by @msx752 in #109
  • Release a Redis node only after its heartbeat stayed missing for a TTL by @msx752 in #107
  • Add a container scenario for the Redis heartbeat after an outage by @msx752 in #110
  • Correct five statements of the guide that said more than the code does by @msx752 in #111
  • Release 1.3.0-beta.2 by @msx752 in #112

Full Changelog: v1.3.0-beta.1...v1.3.0-beta.2