Skip to content

Release v1.14.0

Choose a tag to compare

@github-actions github-actions released this 02 Oct 07:57
ff3ef5a

Restarting a silo with many interrupted jobs no longer floods MongoDB with one burst of storage calls, and brief connection pool exhaustion no longer fails job activations permanently.

Added

  • JobsOptions.MaxConcurrentRehydration (default 8), MaxConcurrentCleanup (default 4) and MaxConcurrentStepStarts (default 16) bound how many jobs are resumed, how many dead jobs are deleted and how many steps of one job are started at once.
  • MongoDBJobsStorageOptions.TransientRetryCount (default 5) and TransientRetryBaseDelay (default 200 ms): job and job step storage retry MongoWaitQueueFullException, TimeoutException, MongoConnectionException and MongoExecutionTimeoutException with exponential backoff and jitter, log each retry, and surface the original exception when the attempts run out. Non-transient errors are never retried.
  • Documentation of the new options and of the MongoDB connection pool settings that matter for jobs (MaxConnectionPoolSize, MaxConnecting, one shared client).

Changed

  • JobsManager.Rehydrate resumes jobs oldest first through a bounded pool and resolves each job grain from the state it already loaded, so it no longer reads every job a second time.
  • JobsManager cleanup of dead jobs deletes through a bounded pool, and starting or resuming a job starts its steps in bounded batches.

Fixed

  • Rehydrating or cleaning up hundreds of jobs after a restart exhausted the MongoDB connection pool wait queue and failed job grain activations; the work is now bounded and transient pool errors are retried (#31)
  • A job that fails while being resumed or deleted no longer stops the other jobs in the same rehydrate or cleanup pass.