Queued outbox circuit breaker

CHANGES.md

@@ -110,6 +110,22 @@ To be released.
     operators distinguish a slow-draining queue from a queue that sees
     less traffic.  [[#316], [#740], [#759]]
 
+ -  Added an outbound delivery circuit breaker for queued outbox delivery.
+    Fedify now tracks consecutive network and HTTP 5xx delivery failures
+    per remote host (including any non-default port), stores the state in
+    the configured `KvStore`, and requeues messages held by an open circuit
+    instead of repeatedly sending to an unreachable server.  The circuit
+    breaker is enabled by default for queued outbox delivery and can be
+    disabled with
+    `circuitBreaker: false`; applications can customize the failure policy,
+    recovery delay, held activity TTL, release interval, and state/drop
+    callbacks.  HTTP 429 responses do not count as circuit failures and
+    `Retry-After` is respected when present.  State changes are exposed
+    through `activitypub.circuit_breaker.state_change` metrics and
+    `activitypub.circuit_breaker.state_change` span events, and expired
+    held activities call the outbox permanent failure handler with
+    `reason: "circuit-breaker-ttl"`.  [[#620], [#778]]
+
  -  Added OpenTelemetry metrics for ActivityPub fanout and activity
     lifecycle events, complementing the per-recipient
     `activitypub.delivery.*` counters and the per-task
@@ -155,10 +171,11 @@ To be released.
     Instruments share an `activitypub.lookup.kind` and (where
     applicable) `activitypub.lookup.result` attribute drawn from small,
     spec-bounded enumerations.  `activitypub.remote.host` records the
-    URL hostname only; `http.response.status_code` is recorded when an
-    HTTP response was observed; `activitypub.cache.enabled` is
-    recorded on the key and document fetch metrics whenever Fedify can
-    confidently report the cache layer's presence.  Key IDs, actor
+    URL host, including any non-default port; `http.response.status_code`
+    is recorded when an HTTP response was observed;
+    `activitypub.cache.enabled` is recorded on the key and document
+    fetch metrics whenever Fedify can confidently report the cache
+    layer's presence.  Key IDs, actor
     IDs, object IDs, JSON-LD context URLs, full URLs, and fediverse
     handles are deliberately excluded so attacker-controlled remotes
     cannot inflate metric cardinality.  The existing
@@ -193,8 +210,9 @@ To be released.
     `webfinger.resource.scheme` is bucketed to a small allow list
     (`acct`, `http`, `https`, `mailto`, or `other`) so an
     attacker-controlled query string cannot inflate metric
-    cardinality; `activitypub.remote.host` records the URL hostname
-    only.  Full resource URIs, lookup URLs, and handle strings are
+    cardinality; `activitypub.remote.host` records the URL host,
+    including any non-default port.  Full resource URIs, lookup URLs,
+    and handle strings are
     deliberately excluded; they remain on the corresponding spans
     (`webfinger.lookup`, `webfinger.handle`,
     `activitypub.get_actor_handle`) for trace-level investigation.
@@ -221,6 +239,7 @@ To be released.
 [#316]: https://github.com/fedify-dev/fedify/issues/316
 [#418]: https://github.com/fedify-dev/fedify/issues/418
 [#619]: https://github.com/fedify-dev/fedify/issues/619
+[#620]: https://github.com/fedify-dev/fedify/issues/620
 [#735]: https://github.com/fedify-dev/fedify/issues/735
 [#736]: https://github.com/fedify-dev/fedify/issues/736
 [#737]: https://github.com/fedify-dev/fedify/issues/737
@@ -241,6 +260,7 @@ To be released.
 [#771]: https://github.com/fedify-dev/fedify/pull/771
 [#772]: https://github.com/fedify-dev/fedify/pull/772
 [#777]: https://github.com/fedify-dev/fedify/pull/777
+[#778]: https://github.com/fedify-dev/fedify/pull/778
 
 ### @fedify/fixture
 

docs/.vitepress/config.mts

@@ -145,6 +145,7 @@ const MANUAL = {
     { text: "Pragmatics", link: "/manual/pragmatics.md" },
     { text: "Key–value store", link: "/manual/kv.md" },
     { text: "Message queue", link: "/manual/mq.md" },
+    { text: "Circuit breaker", link: "/manual/circuit-breaker.md" },
     { text: "Integration", link: "/manual/integration.md" },
     { text: "Migration", link: "/manual/migrate.md" },
     { text: "Relay", link: "/manual/relay.md" },

docs/manual/circuit-breaker.md

@@ -0,0 +1,175 @@
+Circuit breaker
+===============
+
+*This API is available since Fedify 2.3.0.*
+
+Fedify's outbound delivery circuit breaker protects queued ActivityPub
+delivery from repeatedly hammering a remote server that is down or returning
+server errors.  It applies to queued outbox delivery: activities delivered
+through a configured `MessageQueue` are tracked per remote inbox host, and an
+unhealthy host can temporarily hold further deliveries until a recovery probe
+is due.
+
+
+Enabling and disabling
+----------------------
+
+The circuit breaker is enabled by default for queued outbox delivery.  To
+disable it, pass `circuitBreaker: false` to `createFederation()`:
+
+~~~~ typescript
+import { createFederation } from "@fedify/fedify";
+
+const federation = createFederation<void>({
+  kv,
+  queue,
+  circuitBreaker: false,
+});
+~~~~
+
+To customize the defaults, pass a `CircuitBreakerOptions` object:
+
+~~~~ typescript
+import { createFederation } from "@fedify/fedify";
+
+const federation = createFederation<void>({
+  kv,
+  queue,
+  circuitBreaker: {
+    failureThreshold: 5,
+    failureWindow: { minutes: 10 },
+    recoveryDelay: { minutes: 30 },
+    heldActivityTtl: { days: 7 },
+    releaseInterval: { seconds: 1 },
+  },
+});
+~~~~
+
+The default policy opens a remote host's circuit after five consecutive
+counted failures within ten minutes.  When the circuit is open, Fedify
+requeues affected outbox messages instead of sending them.  After the
+`recoveryDelay`, one message is allowed through as a half-open probe.  If it
+succeeds, the circuit closes; if it fails, the circuit opens again.
+While the probe is in flight, other held messages continue to be requeued at
+`releaseInterval`.  If the worker running the probe stops before recording a
+success or failure, Fedify treats the half-open probe as stale after another
+`recoveryDelay` and allows a replacement probe.
+
+
+What counts as a failure
+------------------------
+
+Fedify counts these delivery failures toward the circuit:
+
+ -  network errors, including failed `fetch()` calls
+ -  HTTP 5xx responses from the remote inbox
+
+Fedify does not count these responses as circuit failures:
+
+ -  HTTP 429 responses; the `Retry-After` header is respected when present
+ -  HTTP 4xx responses that are not configured as permanent delivery failures
+ -  configured permanent delivery failures, such as `404` or `410` by default
+
+Any reachable HTTP 4xx response clears the consecutive failure history for
+that host because it proves the remote server can be reached.
+
+
+Custom failure policy
+---------------------
+
+You can replace the numeric threshold/window policy with a callback.  The
+callback receives the full consecutive failure timestamp list for the remote
+host and returns whether the circuit should open:
+
+~~~~ typescript
+const federation = createFederation<void>({
+  kv,
+  queue,
+  circuitBreaker: {
+    failure(timestamps) {
+      return timestamps.length >= 10;
+    },
+  },
+});
+~~~~
+
+The callback form is mutually exclusive with `failureThreshold` and
+`failureWindow`.
+
+
+Held activity expiry
+--------------------
+
+Activities held by an open circuit are requeued until the remote host recovers
+or the held activity exceeds `heldActivityTtl`, which defaults to seven days.
+When a held activity expires, Fedify drops it, records it as an abandoned
+outbox activity, calls `circuitBreaker.onActivityDrop` when configured, and
+calls the outbox permanent failure handler with
+`reason: "circuit-breaker-ttl"`.
+
+~~~~ typescript
+const federation = createFederation<void>({
+  kv,
+  queue,
+  circuitBreaker: {
+    onActivityDrop(remoteHost, details) {
+      console.warn("Dropped held activity", {
+        remoteHost,
+        inbox: details.inbox.href,
+        activityId: details.activityId,
+        heldSince: details.heldSince.toString(),
+      });
+    },
+  },
+});
+
+federation.setOutboxPermanentFailureHandler((_ctx, failure) => {
+  if (failure.reason === "circuit-breaker-ttl") {
+    // The remote host did not recover before the held activity expired.
+    return;
+  }
+
+  // Existing HTTP permanent-failure handling, such as 404 or 410 cleanup.
+});
+~~~~
+
+
+Storage and concurrency
+-----------------------
+
+Circuit state is stored in the configured `KvStore` under the
+`["_fedify", "circuit", remoteHost]` key prefix by default.  The stored value
+has this shape:
+
+~~~~ typescript
+{
+  state: "closed" | "open" | "half-open",
+  failures: string[],
+  opened?: string,
+}
+~~~~
+
+For multi-worker deployments, use a `KvStore` implementation that supports
+`cas()` so competing workers do not overwrite each other's state transitions.
+Fedify still works without CAS, but it logs a warning because concurrent
+workers can race when opening or closing the same host's circuit.
+
+
+Observability
+-------------
+
+State changes are emitted through the `onStateChange` callback and through
+OpenTelemetry:
+
+ -  `activitypub.circuit_breaker.state_change` counter with
+    `activitypub.remote.host` and `activitypub.circuit_breaker.state`
+ -  `activitypub.circuit_breaker.state_change` span event on the queued
+    outbox worker span with the previous and new state
+ -  `activitypub.circuit_breaker.held` span event on the queued outbox worker
+    span when an open circuit holds a delivery
+
+The circuit breaker deliberately records only the remote host, not full inbox
+URLs, actor IDs, or activity IDs, to keep metric cardinality bounded.  For the
+full metric and span attribute lists, see the [OpenTelemetry] manual.
+
+[OpenTelemetry]: ./opentelemetry.md