Skip to content

Release v19.80.0

Choose a tag to compare

@github-actions github-actions released this 17 Oct 18:31
fac8444

Observer replay, catch-up and recovery jobs are now bounded, retried and reconciled instead of piling work onto a single observer until it times out and stays Running across restarts.

Added

  • Stopping or resuming a job now answers whether the job took the request and why not (StopJobResponse and ResumeJobResponse with an Accepted flag and a JobCommandRefusal), through the contracts, the .NET client (IJobs.Stop and IJobs.Resume return a JobCommandResult) and the Workbench, which reports a refusal as an error. The proto changes are additive. (part of #4991)
  • RetryFailedPartitions command and gRPC operation that retries every failed partition of an observer through per-partition jobs started a bounded batch at a time, without reopening a concluded replay job. (part of #4991)

Changed

  • Cratis.Orleans is updated to 1.23.0.
  • A step that times out is run again, up to five attempts, before its partition is reported as failed, so a busy observer no longer turns into a cascade of failed partitions. (part of #4991)
  • A failed job step records one short message instead of a multi-kilobyte stack trace. (part of #4644)
  • Replay keeps at most 16 partition steps active at once instead of starting all of them. (part of #4991)
  • A job recorded as Running whose steps all concluded more than two minutes ago is no longer treated as running, so observers replay starts a real job and a replaying observer without a live job is routed again. (part of #4991)
  • The replay-complete notification to the observer is retried instead of being fired once and forgotten. (part of #4991)
  • Observers coalesce handled-event counts and failure follow-up writes, which takes load off the observer grain. (part of #4991)
  • The kernel statistics projections are no longer fully replayed on every restart, and superseded read model generations are deleted after a replay. (part of #4692) (part of #4644)

Fixed

  • Catch-up that produces no work no longer loops between Routing, CatchUp and CaughtUp without backoff. (#4592)