Skip to content

Release v19.4.3

Choose a tag to compare

@github-actions github-actions released this 23 Sep 20:02
6b41214

Fixed

  • An observer could be left permanently believing it was preparing catch-up — dropping every live event, never advancing, and eventually quarantined — when its catch-up job concluded without having prepared steps of its own

CatchUp raises a preparing flag before asking for a catch-up job, and only RegisterCatchingUpPartitions lowered it again. That is reached exclusively when a brand-new job prepares its steps. Three other outcomes conclude catch-up without ever going through PrepareSteps:

  • a job that was already running,
  • a stopped job that was merely resumed,
  • a job found with every step already completed, which is finalized rather than resumed.

All three call back through CaughtUp, and CaughtUp did not lower the flag. A raised flag makes Handle drop every live event and makes Observing skip its missed-events check, so the observer never observes anything again — and being kept alive, it is never reactivated out of it.

The watchdog's stranded-preparation rescue then clears the flag and re-routes, catch-up concludes exactly the same way on the next tick, and after five consecutive attempts the observer is quarantined for a strand it could not escape.

Observed in a production store on 19.4.1 — which already carries the rescue and revive fixes, so this is the cause upstream of them. Over 20 minutes on one silo:

 259  Job was left running with all of its steps already completed - finalizing it instead of resuming
 101  Watchdog detected that the observer is preparing catch-up with no catch-up job to finish the preparation
  16  Observer has failed to recover from a stranded catch-up preparation 6 times in a row (max 5). Quarantining

100 of 185 observers were more than 500 events behind the tail, 28 quarantined, and the worst frozen ~35,000 events back. Read models served stale state indefinitely with no failed partitions and nothing to indicate a problem.

CaughtUp now lowers the flag and resets the consecutive-strand counter, since catch-up genuinely concluded.

Verified non-vacuous: the new assertion fails with the two lines removed.