Release v1.16.3

Changed

• The jobs documentation now explains what happens to a job step's state when the step succeeds, fails, is stopped or is removed, that PerformStep works on a snapshot of the state, and how to deliver outside input such as worker callbacks to a running job through the job's own grain. See https://github.com/Cratis/Orleans/blob/main/Documentation/concepts/jobs.md (#66)

Fixed

• A job step whose job is removed now has its stored state deleted. It used to be saved again and stay until the job's own state was cleared (#71)

Release v1.16.2

Fixed

• An explicit IJobsManager.Rehydrate() call that arrives just as a background drain finishes now starts a new pass. It used to be skipped, so newly interrupted jobs waited for the next call or a host restart (#68)
• A job step stopped before PerformStep runs (while waiting for a throttle slot, right after acquiring one, or before its work starts) now reports itself as stopped to its job exactly once, so the job completes its stop and runs OnStopped. Job steps that report this outcome themselves from a PerformWork override must remove that code, or the job counts the stop twice. A step whose grain deactivates without a stop request, for example on silo shutdown, still stays scheduled and resumes after restart (#69)

Release v1.16.1

Fixed

• Stopping a job step while it waits for a throttle slot or performs its work now ends in a normal cancellation, instead of a NullReferenceException from JobStep.PerformWork. Code that caught NullReferenceException around base.PerformWork can drop that catch (#65)
• A job step that was stopped can be started again in the same activation; Start used to fail with an unknown error (#65)

Release v1.16.0

Jobs are now placed with resource-optimized placement, so a job started through any replica activates on the least-loaded silo that hosts that job type instead of a randomly chosen one.

Changed

• Job grains use resource-optimized placement (CPU, memory and activation count) among the silos that host the job type. Cratis/StudioIssues#345

Release v1.15.3

Fixed

• Trace context now propagates across grain calls in silos hosted with AddCratisOrleans, while tenant context remains separate from activity baggage. (#47)

Release v1.15.2

Fixed

• Stopping or removing a job no longer stalls for the grain call timeout (30 seconds by default) when one of its steps is reporting a failure or cancellation to the job at the same moment. The stop now reaches the step while it is reporting, so callers waiting on IJobsManager.Stop or Delete, such as a Chronicle observer pausing its jobs when a client disconnects, are no longer held up behind it. The step's outcome is still recorded on the job. See Cratis/Chronicle#4559 for the Chronicle symptoms.

Release v1.15.1

Fixed

• Resuming a stopped job whose steps had all already finished now completes the job and runs its completion, instead of leaving it stopped while reporting it as completed. A Chronicle observer whose catch-up job was stopped by a client disconnect was otherwise left preparing catch-up until it was quarantined. (part of Cratis/Chronicle#4363)
• A job step that has reported its outcome is no longer handed back to the next resume as a step left to run when unsubscribing it fails.

Release v1.15.0

Fixed

• Jobs storage now pages type, status, and age-filtered results, while bounded retention removes old immutable terminal jobs and their steps without touching active jobs. (#53)

Release v1.14.1

Fixed

• Job rehydration returns once the interrupted jobs are loaded and resumes them in the background, so a large backlog no longer times out a silo's startup. A job stopped before its turn is not resumed, and resuming stops when the jobs manager deactivates. (#51)

Release v1.14.0

Restarting a silo with many interrupted jobs no longer floods MongoDB with one burst of storage calls, and brief connection pool exhaustion no longer fails job activations permanently.

Added

• JobsOptions.MaxConcurrentRehydration (default 8), MaxConcurrentCleanup (default 4) and MaxConcurrentStepStarts (default 16) bound how many jobs are resumed, how many dead jobs are deleted and how many steps of one job are started at once.
• MongoDBJobsStorageOptions.TransientRetryCount (default 5) and TransientRetryBaseDelay (default 200 ms): job and job step storage retry MongoWaitQueueFullException, TimeoutException, MongoConnectionException and MongoExecutionTimeoutException with exponential backoff and jitter, log each retry, and surface the original exception when the attempts run out. Non-transient errors are never retried.
• Documentation of the new options and of the MongoDB connection pool settings that matter for jobs (MaxConnectionPoolSize, MaxConnecting, one shared client).

Changed

• JobsManager.Rehydrate resumes jobs oldest first through a bounded pool and resolves each job grain from the state it already loaded, so it no longer reads every job a second time.
• JobsManager cleanup of dead jobs deletes through a bounded pool, and starting or resuming a job starts its steps in bounded batches.

Fixed

• Rehydrating or cleaning up hundreds of jobs after a restart exhausted the MongoDB connection pool wait queue and failed job grain activations; the work is now bounded and transient pool errors are retried (#31)
• A job that fails while being resumed or deleted no longer stops the other jobs in the same rehydrate or cleanup pass.