Skip to content

feat: implement retry policy for jobs#433

Merged
niteshpurohit merged 4 commits intomainfrom
feat/retry-and-backoff
Apr 10, 2026
Merged

feat: implement retry policy for jobs#433
niteshpurohit merged 4 commits intomainfrom
feat/retry-and-backoff

Conversation

@niteshpurohit
Copy link
Copy Markdown
Member

  • Introduced a new Karya::RetryPolicy class to manage retry and backoff behavior for jobs.
  • Added configuration options for maximum attempts, base delay, multiplier, and optional maximum delay.
  • Integrated retry policy into the worker configuration, allowing per-job overrides.
  • Updated job attributes to include retry policy and next retry timestamp.
  • Enhanced worker behavior to handle retry logic, transitioning jobs to a retry_pending state when necessary.
  • Implemented tests to validate retry policy functionality and integration with job processing.
  • Updated documentation to reflect new retry policy features and usage.

closes: #40

- Introduced a new `Karya::RetryPolicy` class to manage retry and backoff behavior for jobs.
- Added configuration options for maximum attempts, base delay, multiplier, and optional maximum delay.
- Integrated retry policy into the worker configuration, allowing per-job overrides.
- Updated job attributes to include retry policy and next retry timestamp.
- Enhanced worker behavior to handle retry logic, transitioning jobs to a `retry_pending` state when necessary.
- Implemented tests to validate retry policy functionality and integration with job processing.
- Updated documentation to reflect new retry policy features and usage.
@niteshpurohit niteshpurohit self-assigned this Apr 10, 2026
Copilot AI review requested due to automatic review settings April 10, 2026 04:27
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements core retry/backoff support for Karya jobs by introducing an immutable Karya::RetryPolicy, persisting retry metadata on jobs (retry_policy, next_retry_at), and teaching the in-memory queue store + worker to transition failed executions into a retry_pending waiting state and lazily re-promote due retries back to queued.

Changes:

  • Add Karya::RetryPolicy (validation, deterministic exponential backoff, optional max-delay clamp) plus RBS + unit tests.
  • Extend Karya::Job/attributes with retry_policy and next_retry_at, including transition support and specs.
  • Integrate retries into Worker and QueueStore::InMemory (new retry_pending workflow, due-retry promotion during maintenance) plus integration tests and docs updates.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
docs/pages/runtime/job-model.md Documents retry metadata and retry_pending lifecycle semantics in the job model.
docs/pages/reliability/retries.md Describes implemented retry behavior vs deferred follow-on work.
core/karya/spec/karya/worker_spec.rb Adds worker configuration validation and retry policy exposure coverage.
core/karya/spec/karya/worker_integration_spec.rb Adds end-to-end retry behavior tests (worker default vs job override).
core/karya/spec/karya/retry_policy_spec.rb New unit tests for retry policy validation and backoff math.
core/karya/spec/karya/queue_store/in_memory_state_helpers_spec.rb Adds coverage for retry-pending state tracking behavior.
core/karya/spec/karya/queue_store/in_memory_execution_spec.rb Adds execution failure → retry_pending behavior and promotion tests.
core/karya/spec/karya/job/attributes_spec.rb Validates new job attributes (retry_policy, next_retry_at).
core/karya/spec/karya/job_spec.rb Ensures job immutability and transitions preserve/override retry metadata.
core/karya/sig/karya/worker/configuration.rbs Adds retry_policy to worker configuration typing.
core/karya/sig/karya/worker.rbs Types worker retry_policy and updated failure path signature.
core/karya/sig/karya/retry_policy.rbs New RBS for Karya::RetryPolicy and error type.
core/karya/sig/karya/queue_store/in_memory.rbs Extends queue store typing for retry policy + retry_pending tracking.
core/karya/sig/karya/queue_store/base.rbs Updates fail_execution signature to accept optional retry policy.
core/karya/sig/karya/job/attributes.rbs Types new job attribute fields in the canonical hash.
core/karya/sig/karya/job.rbs Types job retry_policy and next_retry_at plus transition signature changes.
core/karya/sig/karya.rbs Adds retry_policy type alias and allows it in option/job attribute unions.
core/karya/lib/karya/worker/configuration.rb Adds validated retry_policy support to worker configuration.
core/karya/lib/karya/worker.rb Uses effective retry policy (job override > worker default) on failure.
core/karya/lib/karya/retry_policy.rb Implements validated, immutable retry policy behavior.
core/karya/lib/karya/queue_store/in_memory/store_state.rb Adds retry-pending job ID tracking in in-memory store state.
core/karya/lib/karya/queue_store/in_memory/retry_support.rb Adds due-retry promotion and retry_pending transition helpers.
core/karya/lib/karya/queue_store/in_memory/reserve_selection_support.rb Extracts reserve-selection logic into a support module.
core/karya/lib/karya/queue_store/in_memory/execution_recovery.rb Extracts execution recovery into a dedicated class file.
core/karya/lib/karya/queue_store/in_memory.rb Wires retry support into in-memory store failure and maintenance paths.
core/karya/lib/karya/queue_store/base.rb Updates abstract fail_execution signature to accept retry policy.
core/karya/lib/karya/job/attributes.rb Normalizes/validates new retry attributes for jobs.
core/karya/lib/karya/job.rb Persists retry metadata in canonical job structs and transitions.
core/karya/lib/karya.rb Requires karya/retry_policy.
core/karya/.reek.yml Updates reek exclusions for new/expanded APIs and helpers.

- Added an index to track retry pending job IDs to prevent duplication.
- Updated the `register_retry_pending` method to utilize the index for efficient checks.
- Modified the corresponding test to ensure no duplicates are registered.
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 2 comments.

- Included 'retry_policy' dependency in worker.rb to manage job retries effectively.
- Introduced '@retry_pending_job_ids_index' in in_memory.rbs to track the status of pending jobs.
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 1 comment.

- Adjusted expectations for job retry attempts in the worker integration tests.
- Ensured that the first retry attempt is correctly set to 1 and the second to 2.
- This change improves the accuracy of the test cases related to job retry behavior.
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated no new comments.

@niteshpurohit niteshpurohit merged commit d1376ab into main Apr 10, 2026
26 checks passed
@niteshpurohit niteshpurohit deleted the feat/retry-and-backoff branch April 10, 2026 04:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement retry and backoff behavior

2 participants