Retry and Resilience Patterns

Retry and Resilience Patterns

Overview

This page describes the retry strategies and resilience patterns used by phoss-ap for outbound AS4 sending and inbound forwarding, including the per-C3 circuit breaker for outbound communication.

---

Outbound Sending Retry

When an AS4 sending attempt fails, the outbound transaction is marked as failed and scheduled for retry by the periodic retry scheduler.

Retry Strategy

The retry strategy is configurable via Java interfaces and the following properties:

• retry.sending.max-attempts — Maximum number of attempts before permanent failure
• retry.sending.initial-backoff.ms — Initial backoff interval
• retry.sending.backoff-multiplier — Exponential backoff multiplier
• retry.sending.max-backoff.ms — Maximum backoff cap
• retry.scheduler.interval.ms — How often the scheduler checks for eligible retries

Each failed attempt creates a new outbound_sending_attempt row. The next_retry_dt on the parent outbound_transaction is computed using the backoff strategy and stored in the database, so the scheduler can efficiently query WHERE status = 'failed' AND next_retry_dt <= NOW().

Multi-Instance Safety

The retry scheduler uses SELECT ... FOR UPDATE SKIP LOCKED when picking up eligible transactions. This ensures that when multiple AP instances run concurrently, each transaction is retried by exactly one instance. No leader election is needed — all instances run the scheduler, and the database coordinates work distribution.

phase4 Internal HTTP Retry Disabled

phase4 offers its own internal HTTP retry mechanism (HttpRetrySettings). In phoss-ap this is set to 0 (disabled), because all retry logic is handled at the application level by the phoss-ap retry scheduler. This ensures a clean separation: each phoss-ap retry attempt creates exactly one outbound_sending_attempt row with its own AS4 Message ID.

The circuit breaker (see below) also operates at the application level, gating whether a new sending attempt should be initiated at all.

---

Inbound Forwarding Retry

When forwarding a received document to the Receiver Backend fails, the inbound transaction is marked as forward_failed and scheduled for retry.

Retry Strategy

Analogous to outbound, configured via:

• retry.forwarding.max-attempts — Maximum number of attempts before permanent failure
• retry.forwarding.initial-backoff.ms — Initial backoff interval
• retry.forwarding.backoff-multiplier — Exponential backoff multiplier
• retry.forwarding.max-backoff.ms — Maximum backoff cap

Each failed attempt creates a new inbound_forwarding_attempt row. The next_retry_dt on the parent inbound_transaction is updated accordingly.

Multi-Instance Safety

Same SKIP LOCKED approach as outbound retry — safe for concurrent instances without coordination.

---

Circuit Breaker for Outbound AS4 Sending

Motivation

When a remote AP (C3) is down or unreachable, repeatedly attempting to send AS4 messages wastes resources and delays processing of messages to other destinations. A per-C3 circuit breaker prevents this by temporarily blocking send attempts to an unhealthy endpoint, allowing it time to recover.

Design

The circuit breaker is implemented in-memory using a library such as Failsafe and is keyed by the C3 AP endpoint URL (resolved from SMP lookup), not by receiver participant ID, since multiple participants may share the same AP endpoint.

State transitions follow the standard circuit breaker pattern:

CLOSED  ──(failure threshold reached)──>  OPEN
OPEN    ──(timeout expires)──>            HALF_OPEN
HALF_OPEN ──(probe succeeds)──>           CLOSED
HALF_OPEN ──(probe fails)──>              OPEN

• Closed: Normal operation. Failures are counted.
• Open: All send attempts to this C3 are blocked immediately. The transaction stays in failed status with next_retry_dt set — it will be retried later when the breaker transitions to half-open.
• Half-open: A limited number of probe attempts are allowed through. If they succeed, the breaker closes. If they fail, it reopens.

Configuration

Property	Description	Default
circuit-breaker.failure-threshold	Number of consecutive failures before opening	5
circuit-breaker.open-duration.ms	How long the breaker stays open before transitioning to half-open	60000 (1 minute)
circuit-breaker.half-open-max-attempts	Number of probe attempts allowed in half-open state	1

Integration with phase4

The circuit breaker integrates with phase4's existing extension points — no phase4 source code modifications are required. The sendMessageAndCheckForReceipt() method provides two key feedback channels:

1. A Consumer<Phase4Exception> parameter for exception handling
2. An IAS4SignalMessageConsumer callback for inspecting the AS4 Signal Message (receipt or error)

Combined with IAS4SenderInterrupt for pre-send gating, the full integration uses three phase4 interfaces:

1. IAS4SenderInterrupt — Pre-send gate

Called after all message preparation is done but before the actual AS4 HTTP transmission. If the circuit breaker for the target C3 is open, it returns EContinue.BREAK to prevent sending.

// In phoss-ap-core outbound orchestration:
Phase4PeppolSender.builder ()
    // ... standard configuration ...
    .senderInterrupt (() -> {
        if (circuitBreakerManager.isOpen (c3EndpointURL))
          return EContinue.BREAK;
        return EContinue.CONTINUE;
    })

When the interrupt blocks sending, the transaction remains in failed status. The retry scheduler will attempt it again after next_retry_dt.

2. IAS4SignalMessageConsumer — Success/error detection via AS4 Signal Message

Set via .signalMsgConsumer() on the builder. Called when the synchronous AS4 response (Receipt or Error) is received. The Ebms3SignalMessage can be inspected to determine whether the response is a valid receipt or an AS4 error signal. On success, the circuit breaker records a successful call.

    .signalMsgConsumer ((aSignalMsg, aMessageMetadata, aState) -> {
        // A receipt was received — record success
        if (aSignalMsg.getReceipt () != null)
          circuitBreakerManager.recordSuccess (c3EndpointURL);
        else {
          // Sending failed — record failure in circuit breaker
          circuitBreakerManager.recordFailure (c3EndpointURL);
          // Also log or store the exception for error_details
          LOGGER.error ("AS4 error received from " + c3EndpointURL, ex);
        }          
    })

3. Consumer<Phase4Exception> — Failure detection via exception

The sendMessageAndCheckForReceipt(Consumer) method accepts a Consumer<Phase4Exception> that is invoked when an exception occurs during sending (e.g., connection refused, timeout, TLS errors). This is the failure signal for the circuit breaker.

final EAS4UserMessageSendResult eResult =
    aBuilder.sendMessageAndCheckForReceipt (ex -> {
        // Sending failed — record failure in circuit breaker
        circuitBreakerManager.recordFailure (c3EndpointURL);
        // Also log or store the exception for error_details
        LOGGER.error ("AS4 sending failed to " + c3EndpointURL, ex);
    });

The returned EAS4UserMessageSendResult is then used by phoss-ap to determine the overall outcome and update the outbound_sending_attempt and outbound_transaction accordingly.

Phase4 Extension Points Summary

Phase4PeppolSender.builder()
    │
    ├─ .senderInterrupt()            ← Circuit breaker gate (pre-send)
    ├─ .signalMsgConsumer()          ← Success tracking (receipt received)
    │
    └─ sendMessageAndCheckForReceipt(exceptionConsumer)
           │
           ├─ HTTP transmission
           │      │
           │      ├─ On success or failure → IAS4SignalMessageConsumer called
           │      │                  → recordSuccess()
           │      │
           │      └─ On failure → Consumer<Phase4Exception> called
           │                        → recordFailure()
           │
           └─ Returns EAS4UserMessageSendResult

In-Memory Design

The circuit breaker state is held purely in-memory (ConcurrentHashMap<String, CircuitBreaker> keyed by C3 endpoint URL). This means:

• State is lost on AP restart — this is intentional, as a restart is itself a recovery event.
• No database persistence overhead for circuit breaker state.
• Thread-safe for concurrent sending to different C3 endpoints.
• In a multi-instance deployment, each instance maintains its own independent circuit breaker state. This is acceptable — each instance protects itself, and different instances may have different views of a C3 endpoint's health.

Interaction with Application-Level Retry

When the circuit breaker blocks a send attempt via IAS4SenderInterrupt:

1. The sending attempt is not made (no outbound_sending_attempt row is created).
2. The outbound_transaction remains in failed status.
3. The next_retry_dt is recalculated using the backoff strategy and updated.
4. The retry scheduler will try again after the backoff period — by which time the breaker may have transitioned to half-open.

This ensures that the circuit breaker and the retry scheduler work together without conflicting.

---

Forwarding Circuit Breaker (Future Consideration)

A similar circuit breaker pattern could be applied to inbound forwarding (per Receiver Backend endpoint). Since the forwarding mode is configured per AP instance (single endpoint), this would be a single circuit breaker rather than a per-endpoint map. This is noted as a future consideration and not part of the initial implementation.

---

Module Placement

Concern	Module
Circuit breaker manager (in-memory state, Failsafe integration)	phoss-ap-core
Retry strategy interface	phoss-ap-api
Retry scheduler	phoss-ap-core
Circuit breaker configuration properties	phoss-ap-webapp (application.properties)

The Failsafe library dependency is added to phoss-ap-core only.