feat: implement retry policy for jobs#433
Merged
niteshpurohit merged 4 commits intomainfrom Apr 10, 2026
Merged
Conversation
- Introduced a new `Karya::RetryPolicy` class to manage retry and backoff behavior for jobs. - Added configuration options for maximum attempts, base delay, multiplier, and optional maximum delay. - Integrated retry policy into the worker configuration, allowing per-job overrides. - Updated job attributes to include retry policy and next retry timestamp. - Enhanced worker behavior to handle retry logic, transitioning jobs to a `retry_pending` state when necessary. - Implemented tests to validate retry policy functionality and integration with job processing. - Updated documentation to reflect new retry policy features and usage.
There was a problem hiding this comment.
Pull request overview
Implements core retry/backoff support for Karya jobs by introducing an immutable Karya::RetryPolicy, persisting retry metadata on jobs (retry_policy, next_retry_at), and teaching the in-memory queue store + worker to transition failed executions into a retry_pending waiting state and lazily re-promote due retries back to queued.
Changes:
- Add
Karya::RetryPolicy(validation, deterministic exponential backoff, optional max-delay clamp) plus RBS + unit tests. - Extend
Karya::Job/attributes withretry_policyandnext_retry_at, including transition support and specs. - Integrate retries into
WorkerandQueueStore::InMemory(newretry_pendingworkflow, due-retry promotion during maintenance) plus integration tests and docs updates.
Reviewed changes
Copilot reviewed 30 out of 30 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/pages/runtime/job-model.md | Documents retry metadata and retry_pending lifecycle semantics in the job model. |
| docs/pages/reliability/retries.md | Describes implemented retry behavior vs deferred follow-on work. |
| core/karya/spec/karya/worker_spec.rb | Adds worker configuration validation and retry policy exposure coverage. |
| core/karya/spec/karya/worker_integration_spec.rb | Adds end-to-end retry behavior tests (worker default vs job override). |
| core/karya/spec/karya/retry_policy_spec.rb | New unit tests for retry policy validation and backoff math. |
| core/karya/spec/karya/queue_store/in_memory_state_helpers_spec.rb | Adds coverage for retry-pending state tracking behavior. |
| core/karya/spec/karya/queue_store/in_memory_execution_spec.rb | Adds execution failure → retry_pending behavior and promotion tests. |
| core/karya/spec/karya/job/attributes_spec.rb | Validates new job attributes (retry_policy, next_retry_at). |
| core/karya/spec/karya/job_spec.rb | Ensures job immutability and transitions preserve/override retry metadata. |
| core/karya/sig/karya/worker/configuration.rbs | Adds retry_policy to worker configuration typing. |
| core/karya/sig/karya/worker.rbs | Types worker retry_policy and updated failure path signature. |
| core/karya/sig/karya/retry_policy.rbs | New RBS for Karya::RetryPolicy and error type. |
| core/karya/sig/karya/queue_store/in_memory.rbs | Extends queue store typing for retry policy + retry_pending tracking. |
| core/karya/sig/karya/queue_store/base.rbs | Updates fail_execution signature to accept optional retry policy. |
| core/karya/sig/karya/job/attributes.rbs | Types new job attribute fields in the canonical hash. |
| core/karya/sig/karya/job.rbs | Types job retry_policy and next_retry_at plus transition signature changes. |
| core/karya/sig/karya.rbs | Adds retry_policy type alias and allows it in option/job attribute unions. |
| core/karya/lib/karya/worker/configuration.rb | Adds validated retry_policy support to worker configuration. |
| core/karya/lib/karya/worker.rb | Uses effective retry policy (job override > worker default) on failure. |
| core/karya/lib/karya/retry_policy.rb | Implements validated, immutable retry policy behavior. |
| core/karya/lib/karya/queue_store/in_memory/store_state.rb | Adds retry-pending job ID tracking in in-memory store state. |
| core/karya/lib/karya/queue_store/in_memory/retry_support.rb | Adds due-retry promotion and retry_pending transition helpers. |
| core/karya/lib/karya/queue_store/in_memory/reserve_selection_support.rb | Extracts reserve-selection logic into a support module. |
| core/karya/lib/karya/queue_store/in_memory/execution_recovery.rb | Extracts execution recovery into a dedicated class file. |
| core/karya/lib/karya/queue_store/in_memory.rb | Wires retry support into in-memory store failure and maintenance paths. |
| core/karya/lib/karya/queue_store/base.rb | Updates abstract fail_execution signature to accept retry policy. |
| core/karya/lib/karya/job/attributes.rb | Normalizes/validates new retry attributes for jobs. |
| core/karya/lib/karya/job.rb | Persists retry metadata in canonical job structs and transitions. |
| core/karya/lib/karya.rb | Requires karya/retry_policy. |
| core/karya/.reek.yml | Updates reek exclusions for new/expanded APIs and helpers. |
- Added an index to track retry pending job IDs to prevent duplication. - Updated the `register_retry_pending` method to utilize the index for efficient checks. - Modified the corresponding test to ensure no duplicates are registered.
- Included 'retry_policy' dependency in worker.rb to manage job retries effectively. - Introduced '@retry_pending_job_ids_index' in in_memory.rbs to track the status of pending jobs.
- Adjusted expectations for job retry attempts in the worker integration tests. - Ensured that the first retry attempt is correctly set to 1 and the second to 2. - This change improves the accuracy of the test cases related to job retry behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Karya::RetryPolicyclass to manage retry and backoff behavior for jobs.retry_pendingstate when necessary.closes: #40