Repository navigation
Release v19.4.7
Fixed
- An observer whose stopped catch-up job refused to resume was left believing catch-up was under way — dropping every live event, never advancing, and eventually quarantined for a strand it could not escape
A stopped catch-up job is offered back to the observer that owns it, and resuming is allowed to refuse: the job was never prepared, or the observer is no longer subscribed. IJobsManager.Resume computed that refusal, logged it, and returned void — so StartOrResumeObserverJobFor reported the job's id as though something had taken it.
Nothing had. A refused job stays Stopped: it will not run, will not finalize, and will never report back — and reporting back through CaughtUp is what lowers the preparing flag. While the flag is up, Handle drops every live event and Observing skips its missed-events check, so the observer stops advancing at all. The watchdog then clears the flag and re-routes, catch-up finds the same stopped job, it refuses again, and after five rounds the observer is quarantined.
Found by restoring a production event store locally and reading the kernel's own job records:
CatchUpObserver Stopped x30 <- offered and refused, forever
CatchUpObserver CompletedWithFailures x31
CatchUpObserver Failed x3
Against 185 observers in that store, 100 were more than 500 events behind the tail and 29 were quarantined, with no failed partitions and nothing surfacing the cause. The affected observers were reactors — issue triage, status propagation, branch naming — all stalled at the same position, so the application's automation had silently stopped.
Resume now reports whether the job was taken forward, and a refusal clears the preparation rather than pretending it is under way. A job that fails to start is a different situation and keeps its existing retry-then-quarantine handling — the specs covering that path are untouched and still pass.
Verified non-vacuous: the new assertion fails with the check removed.