Repository navigation
Release v1.14.0
Restarting a silo with many interrupted jobs no longer floods MongoDB with one burst of storage calls, and brief connection pool exhaustion no longer fails job activations permanently.
Added
JobsOptions.MaxConcurrentRehydration(default 8),MaxConcurrentCleanup(default 4) andMaxConcurrentStepStarts(default 16) bound how many jobs are resumed, how many dead jobs are deleted and how many steps of one job are started at once.MongoDBJobsStorageOptions.TransientRetryCount(default 5) andTransientRetryBaseDelay(default 200 ms): job and job step storage retryMongoWaitQueueFullException,TimeoutException,MongoConnectionExceptionandMongoExecutionTimeoutExceptionwith exponential backoff and jitter, log each retry, and surface the original exception when the attempts run out. Non-transient errors are never retried.- Documentation of the new options and of the MongoDB connection pool settings that matter for jobs (
MaxConnectionPoolSize,MaxConnecting, one shared client).
Changed
JobsManager.Rehydrateresumes jobs oldest first through a bounded pool and resolves each job grain from the state it already loaded, so it no longer reads every job a second time.JobsManagercleanup of dead jobs deletes through a bounded pool, and starting or resuming a job starts its steps in bounded batches.
Fixed
- Rehydrating or cleaning up hundreds of jobs after a restart exhausted the MongoDB connection pool wait queue and failed job grain activations; the work is now bounded and transient pool errors are retried (#31)
- A job that fails while being resumed or deleted no longer stops the other jobs in the same rehydrate or cleanup pass.