Skip to content

1.4.0

Choose a tag to compare

@msx752 msx752 released this 12 Nov 17:31
2aeab18

1.4.0 closes the two known gaps in how a chain child without a time of its own is started: it could run twice after an edit, and it could be left Idle for good. No database migration for EF Core, no change of the MongoDB collections. Redis gets one new key. Please read Behaviour changes before deploying: children that an earlier version left Idle are run or skipped after the upgrade.

Fixed

Chain children without a time of their own

  • A child given a time while its parent runs no longer runs twice (#83). The parent's run now starts such a child with a claim on its stored row, and the row refuses the claim once it has a time, is held, or is no longer Idle. Before, the parent's run started the child whatever its row said, while the store handed the same child out at its new time.
  • A child whose start claim fails is not run without its row (#83). Before, a failed write at the child's start was logged and the child ran anyway, with nothing fencing its writes. It now keeps waiting and a Warning names it.
  • A child that was never started is run by the fallback (#84). A child that is Idle, has no time and is not held, and whose parent ended at least 30 seconds ago, is taken by the fallback: it runs as a job of its own and ends DueDone when the parent's outcome lets it run, and is recorded as Skipped otherwise. This covers a node that was killed while it ran the child (EF Core and Redis, after the node's rows are released), a node that stopped between the parent's end and the child's start, and a start claim that failed. Before, nothing took such a row.

Behaviour changes to check before upgrading

  • Children left Idle by earlier versions are decided after the upgrade. Every chain child without a time whose parent has ended is run or skipped by the first fallback pass, however old it is. Delete the ones you do not want to run before upgrading.
  • Such a child starts later than before in the failure cases: 30 seconds plus up to one FallbackIntervalChecker (30 seconds by default) after its parent's end, as a job of its own with its own concurrency slot and priority. It ends DueDone, not Done.
  • A child whose parent's outcome does not let it run becomes Skipped where it stayed Idle.
  • Every fallback pass does one more read. On EF Core it is one query without a new index. On MongoDB and Redis its cost grows with the number of chains whose parent has not run yet: about one second per pass with 100 000 such chains, measured on one machine.
  • Redis: one new key, the hash tq:tt:waiting-children. The first fallback pass of each process fills it from the stored jobs; with very many stored time tickers that first pass takes longer.
  • Log texts changed: the entries that told you to add a new job for a child that did not start now say that the fallback takes it.

Upgrade

From 1.3.1: no step. Upgrade all nodes together where you can. While a 1.3.x node still runs, it starts chain children the old way, so the two fixes above do not hold for the chains it runs, and on Redis a child it stores gets no entry in the new key until a 1.4.0 process starts.

Known limits

  • MongoDB has no release step for a node that stopped: a job, and so also a chain child, that was InProgress when its node was killed stays InProgress (#140).
  • At-least-once: a job whose process ended before its lost outcome was stored can run again.
  • Redis: a node that alone could not reach Redis for about two heartbeat TTLs has its running jobs released (#46).
  • A child skipped by the fallback is followed by what it carries one level per pass: each level is skipped 30 seconds and one pass after the level above it.

What's Changed

  • Start an inline chain child only while its row still waits by @msx752 in #133
  • Do not run an inline chain child whose start claim was not won by @msx752 in #135
  • Let the fallback take a chain child without a time that was never started by @msx752 in #136
  • Docs: the start of a chain child without a time, and the fallback that takes one left waiting by @msx752 in #137
  • Add container scenarios for the start of a chain child without a time by @msx752 in #138
  • Release 1.4.0 by @msx752 in #141

Full Changelog: v1.3.1...v1.4.0