Repository navigation
Release v19.4.3
Fixed
- An observer could be left permanently believing it was preparing catch-up — dropping every live event, never advancing, and eventually quarantined — when its catch-up job concluded without having prepared steps of its own
CatchUp raises a preparing flag before asking for a catch-up job, and only RegisterCatchingUpPartitions lowered it again. That is reached exclusively when a brand-new job prepares its steps. Three other outcomes conclude catch-up without ever going through PrepareSteps:
- a job that was already running,
- a stopped job that was merely resumed,
- a job found with every step already completed, which is finalized rather than resumed.
All three call back through CaughtUp, and CaughtUp did not lower the flag. A raised flag makes Handle drop every live event and makes Observing skip its missed-events check, so the observer never observes anything again — and being kept alive, it is never reactivated out of it.
The watchdog's stranded-preparation rescue then clears the flag and re-routes, catch-up concludes exactly the same way on the next tick, and after five consecutive attempts the observer is quarantined for a strand it could not escape.
Observed in a production store on 19.4.1 — which already carries the rescue and revive fixes, so this is the cause upstream of them. Over 20 minutes on one silo:
259 Job was left running with all of its steps already completed - finalizing it instead of resuming
101 Watchdog detected that the observer is preparing catch-up with no catch-up job to finish the preparation
16 Observer has failed to recover from a stranded catch-up preparation 6 times in a row (max 5). Quarantining
100 of 185 observers were more than 500 events behind the tail, 28 quarantined, and the worst frozen ~35,000 events back. Read models served stale state indefinitely with no failed partitions and nothing to indicate a problem.
CaughtUp now lowers the flag and resets the consecutive-strand counter, since catch-up genuinely concluded.
Verified non-vacuous: the new assertion fails with the two lines removed.