Repository navigation
Release v19.80.0
Observer replay, catch-up and recovery jobs are now bounded, retried and reconciled instead of piling work onto a single observer until it times out and stays Running across restarts.
Added
- Stopping or resuming a job now answers whether the job took the request and why not (
StopJobResponseandResumeJobResponsewith anAcceptedflag and aJobCommandRefusal), through the contracts, the .NET client (IJobs.StopandIJobs.Resumereturn aJobCommandResult) and the Workbench, which reports a refusal as an error. The proto changes are additive. (part of #4991) RetryFailedPartitionscommand and gRPC operation that retries every failed partition of an observer through per-partition jobs started a bounded batch at a time, without reopening a concluded replay job. (part of #4991)
Changed
- Cratis.Orleans is updated to 1.23.0.
- A step that times out is run again, up to five attempts, before its partition is reported as failed, so a busy observer no longer turns into a cascade of failed partitions. (part of #4991)
- A failed job step records one short message instead of a multi-kilobyte stack trace. (part of #4644)
- Replay keeps at most 16 partition steps active at once instead of starting all of them. (part of #4991)
- A job recorded as Running whose steps all concluded more than two minutes ago is no longer treated as running, so
observers replaystarts a real job and a replaying observer without a live job is routed again. (part of #4991) - The replay-complete notification to the observer is retried instead of being fired once and forgotten. (part of #4991)
- Observers coalesce handled-event counts and failure follow-up writes, which takes load off the observer grain. (part of #4991)
- The kernel statistics projections are no longer fully replayed on every restart, and superseded read model generations are deleted after a replay. (part of #4692) (part of #4644)
Fixed
- Catch-up that produces no work no longer loops between Routing, CatchUp and CaughtUp without backoff. (#4592)