Advanced Retry Policies

[abdallahsamabd]

Advanced Retry Policies (#107)

Problem

Praxis currently only retries on TCP connect failures — hardcoded max 3 attempts against the same endpoint, idempotent methods only (GET/HEAD/OPTIONS), 64 KiB body cap. This is insufficient for modern distributed systems where transient failures are common during rolling deploys, GC pauses, or downstream overload.

When a backend returns 503 during a rolling update, healthy replicas exist but the proxy forwards the error directly to the client. Operators are forced to push retry logic into every client application.

Proposal

Extend the retry mechanism to support configurable, policy-driven retries at the cluster and route level:

• Retriable status codes — retry on 502, 503, 504 (not just TCP failures)
• Configurable retry count — per cluster and per route (not hardcoded)
• Exponential backoff — configurable base/max intervals to avoid hammering struggling backends
• Retry budget — cap retries as a percentage of active requests to prevent retry storms
• Alternate-host selection — try a different endpoint on each retry instead of hitting the same dead server
• Per-try timeout — independent of overall request timeout so slow backends don't consume the entire budget
• Non-idempotent opt-in — allow POST/PUT retries on routes that guarantee idempotency (e.g. idempotency keys)

Related:

• Issue: Advanced Retry Policies #107
• Epic: Epic: Resilience and Traffic Management #163
• Proposal PR: docs: add proposal for advanced retry policies #901
• Proposal(how?): docs(proposals): add How? section for advanced retry policies #911

Example configuration (aspirational)

clusters:
  - name: api_backend
    endpoints:
      - "10.0.1.1:8080"
      - "10.0.1.2:8080"
      - "10.0.1.3:8080"
    retry_policy:
      max_retries: 3
      retriable_status_codes: [502, 503, 504]
      per_try_timeout_ms: 2000
      backoff:
        base_interval_ms: 25
        max_interval_ms: 250
      retry_host_selection: avoid_previous

[shaneutt]

maintainer +1