Fedify 2.3.0: OpenTelemetry metrics, delivery circuit breaker, @fedify/backfill, and fedify bench

[dahlia]

Fedify is a TypeScript framework for building ActivityPub servers. It implements federation details such as HTTP Signatures, JSON-LD processing, WebFinger, inbox and outbox routing, and activity delivery.

Fedify 2.3.0 is largely a production observability release. OpenTelemetry metrics land across every major federation path, a monitoring guide and runnable example stack ship alongside them, and a delivery circuit breaker keeps queue workers from endlessly retrying against unreachable servers. Two new packages ship with this version: @fedify/backfill for reconstructing ActivityPub conversations, and @fedify/uri-template for RFC 6570 URI Template handling. The documentation home page has also been rebuilt on VitePress 2.

New documentation home page

fedify.dev now runs on VitePress 2 with a custom landing page built around Fedify's own colors and typography. The old page was the stock VitePress hero plus a one-line feature grid. The new one explains Fedify from the outside in: what it does, which ActivityPub specifications it handles, and where it sits in the broader ecosystem. The page also credits the Sovereign Tech Agency, which is investing in Fedify's continued development. (#809)

OpenTelemetry metrics

Fedify has carried OpenTelemetry tracing since its early releases. What was missing was metrics: counters and histograms you can scrape with Prometheus, alert on, and put in a dashboard. Without them, you could trace individual requests but couldn't easily answer questions like “how many activities failed to deliver this hour?” or “what is my p95 signature verification latency?” Fedify 2.3.0 adds metrics to delivery, inbox processing, HTTP handling, signature verification, queues, collections, WebFinger, actor discovery, and remote document fetches.

To enable metrics, pass a MeterProvider to createFederation():

const federation = createFederation<void>({
  kv: store,
  queue: messageQueue,
  meterProvider: myMeterProvider,
});

Context.meterProvider exposes the same provider inside inbox listeners and outbox handlers for application-level meters. Omitting meterProvider disables all metric collection with no overhead.

Activity delivery and inbox processing

activitypub.delivery.sent and activitypub.delivery.permanent_failure (counters) track per-recipient delivery outcomes. activitypub.inbox.processing.duration (histogram) measures how long inbox listener callbacks take. A new activitypub.delivery.failed span event on queued outbox delivery spans captures the remote host, attempt number, and HTTP status code on every retry and permanent-failure decision.

One small breaking change: the existing activitypub.activity.sent span event now records activitypub.inbox.url and activitypub.activity.id rather than the full activitypub.activity.json payload. FedifySpanExporter and TraceActivityRecord.activityJson adapt accordingly. (#316, #619, #755)

HTTP server metrics

fedify.http.server.request.count (Counter) and fedify.http.server.request.duration (Histogram) cover inbound requests handled by Federation.fetch(). Both carry http.request.method, fedify.endpoint, optional http.response.status_code, and optional fedify.route.template. Raw URLs, query strings, and identifier values are excluded to keep cardinality bounded. (#316, #736, #757)

Signature verification

activitypub.signature.verification.duration measures end-to-end verification time for HTTP Signatures, Linked Data Signatures, and Object Integrity Proofs, including both local key lookup and remote key fetches. activitypub.signature.key_fetch.duration isolates public-key lookup latency separately, so operators can distinguish how much of verification time comes from network round trips versus local computation. Both histograms carry bounded activitypub.signature.kind and result attributes, plus algorithm-specific attributes on the verification histogram. (#316, #737, #769)

Queue task metrics

Six instruments cover Fedify's enqueue and worker boundaries for inbox, outbox, and fanout work: fedify.queue.task.enqueued, fedify.queue.task.started, fedify.queue.task.completed, fedify.queue.task.failed (counters), fedify.queue.task.duration (histogram), and fedify.queue.task.in_flight (up/down counter). They carry fedify.queue.role, fedify.queue.backend (the queue implementation's constructor name), fedify.queue.native_retrial, and activitypub.activity.type when Fedify knows the activity type for a queued message. With MessageQueue.getDepth() (see below), you can tell whether the queue is backed up or whether traffic has simply dropped. (#316, #740, #759)

Collections, fanout, and activity lifecycle

activitypub.collection.request, activitypub.collection.dispatch.duration, activitypub.collection.page.items, and activitypub.collection.total_items cover collection dispatch for both built-in and custom collection routes. Built-in collections are classified as inbox, outbox, following, followers, liked, featured, or featured_tags; custom routes collapse to custom. (#316, #741, #777)

activitypub.fanout.recipients (histogram) records how many recipient inboxes a single fanout produces. activitypub.inbox.activity and activitypub.outbox.activity (counters) classify each activity by processing outcome (queued, processed, retried, rejected, or abandoned). (#316, #742, #770)

WebFinger, actor discovery, and remote fetches

Three counter/histogram pairs cover outgoing WebFinger lookups (webfinger.lookup, webfinger.lookup.duration), incoming WebFinger requests (webfinger.handle, webfinger.handle.duration), and getActorHandle() actor discovery (activitypub.actor.discovery, activitypub.actor.discovery.duration). Each metric records a bounded result value, so you can separate network failures, HTTP errors, and other terminal outcomes without blowing up cardinality. (#316, #739, #772)

activitypub.key.lookup and activitypub.key.lookup.duration track all public-key lookups including signature verification paths. activitypub.document.fetch and activitypub.document.fetch.duration cover every Fedify-wrapped document-loader invocation. activitypub.document.cache records cache hits and misses for kvCache()-backed loaders. activitypub.object.lookup records lookupObject() resolution outcomes. Remote host is recorded as a bounded activitypub.remote.host attribute; full URLs, actor IDs, and fediverse handles are excluded to prevent attacker-controlled cardinality. (#316, #738, #771)

Production monitoring guide and example

The new Production monitoring guide starts from the metric names above and turns them into Prometheus queries, Grafana panels, and alert rules. It covers the checks most production Fedify servers will want first: queue backlog, delivery failures, signature failures, federation health, and circuit breaker state. (#813)

examples/monitoring/ has a local setup you can run: an OpenTelemetry Collector, Prometheus, pre-built Grafana dashboards, alert rule files, synthetic Fedify-shaped metrics, validation checks, and an optional Docker Compose smoke test. (#814)

Circuit breaker

Fedify 2.3.0 adds an outbound delivery circuit breaker for a failure mode most fediverse servers eventually hit. A remote instance goes down, local followers keep generating deliveries to it, and workers spend their retry budget on a host that isn't coming back soon, all while obscuring failures to servers that are actually reachable.

The circuit breaker tracks consecutive network and HTTP 5xx failures per remote host in the KvStore and opens when enough failures accumulate within a time window. While the circuit is open, Fedify holds queued activities for that host instead of trying to deliver them. After the recovery delay, a probe delivery tests whether the server is back; success closes the circuit and releases held activities, failure reopens it.

The circuit breaker is on by default. Pass circuitBreaker: false to disable it, or tune the thresholds:

const federation = createFederation<void>({
  kv: store,
  queue: messageQueue,
  circuitBreaker: {
    failureThreshold: 5,
    failureWindow: Temporal.Duration.from({ minutes: 10 }),
    recoveryDelay: Temporal.Duration.from({ minutes: 30 }),
    onStateChange(host, previous, next) {
      logger.warn(`Circuit breaker for ${host}: ${previous} → ${next}`);
    },
  },
});

429 Too Many Requests responses do not count as circuit failures, and Retry-After is honored when present. Activities held past their TTL (default: 7 days) call the permanent failure handler with reason: "circuit-breaker-ttl". State transitions emit activitypub.circuit_breaker.state_change metrics and span events. (#620, #778)

fedify bench

@fedify/cli gains a fedify bench command for ActivityPub-specific load testing. Generic HTTP tools don't sign inbox requests, understand fanout, or map results to ActivityPub instrumentation; fedify bench does.

Benchmark runs are described by YAML or JSON scenario files validated against a published JSON Schema. Supported scenario types include actor and object document fetches, fanout scenarios that drive a benchmark trigger endpoint and wait for queue drain, failure scenarios that report expected fault outcomes as successes, and mixed scenarios that blend children by weight. An expect block per scenario gates CI runs against latency and throughput thresholds.

fedify bench run --target http://localhost:3000 bench/suite.yaml

fedify bench compare checks out base and head refs into temporary worktrees, starts the benchmark target for each, runs the same suite, and fails when the head regresses beyond --max-regression plus the measured per-run noise band. Scenarios run three times by default, aggregating with median latency and throughput and pessimistic correctness results.

The target server opts in with benchmarkMode: true in createFederation(), which exposes GET /.well-known/fedify/bench/stats for in-process metric snapshots and POST /.well-known/fedify/bench/trigger for driving sendActivity() to benchmark sink recipients. In benchmark mode, allowPrivateAddress defaults to true when built-in loaders are used, and signatureTimeWindow defaults to false. (#744, #783, #784, #785, #786, #791, #795, #801, #802, #804)

@fedify/backfill

@fedify/backfill is for the “quiet fediverse” problem: fetch a reply to a thread and you usually get one post, not the conversation around it. FEP-f228 proposes a solution where conversation owners publish a resolvable context collection containing every post in the thread. Without it, the only option is traversing inReplyTo chains recursively, which is slow, rate-limit-sensitive, and often incomplete.

@fedify/backfill tries the FEP-f228 context-owner approach first (the context collection may contain post-like objects or Create activities) and falls back to reply-tree traversal when no such collection is available. You can configure traversal order, share one safety budget across the whole run, deduplicate posts that multiple paths find, and keep a traversal-local document cache so the same objects don't get fetched repeatedly.

import { backfill } from "@fedify/backfill";

for await (const post of backfill(ctx, note)) {
  await saveToDatabase(post);
}

Contributed by Jiwon Kwon (@sij411). (#275, #779, #801, #807, #816, #820)

Queue depth reporting

MessageQueue now supports an optional getDepth() method returning ready and delayed message counts as a MessageQueueDepth object. InProcessMessageQueue reports both values; ParallelMessageQueue delegates to its wrapped queue when supported. @fedify/amqp, @fedify/mysql, @fedify/postgres, @fedify/redis, and @fedify/sqlite all gain getDepth() implementations.

In benchmarkMode, Fedify exposes the result as a fedify.queue.depth gauge so fedify bench can wait for queue drain after triggering a fanout scenario. (#735, #748)

Fixed-path actor dispatchers

Some applications expose a single instance-level actor rather than per-user actors: a relay at /actor, a single-author blog speaking as itself, a bot at /bot. Previously this required a sentinel identifier in the actor path that leaked into mapHandle() and related callbacks, and placed the canonical URI at something like /users/__instance__ rather than the intended fixed path.

ActorCallbackSetters.mapActorAlias() maps a fixed path to a synthetic identifier inside the existing actor dispatcher:

federation
  .setActorDispatcher("/users/{identifier}", async (ctx, identifier) => {
    if (identifier === "__relay__") {
      return new Service({ id: ctx.getActorUri("__relay__"), /* … */ });
    }
    // normal user lookup
  })
  .mapActorAlias("/actor", "__relay__");

WebFinger lookups for the fixed path work correctly, and the actor's canonical URI is /actor rather than /users/__relay__. (#752, #753)

@fedify/uri-template

Fedify's internal federation routing now uses @fedify/uri-template, a dependency-free RFC 6570 implementation. It expands templates, extracts variables from matched URLs, and verifies that generated routes round-trip correctly. It replaces the previous combination of url-template and uri-template-router. The old Router export from @fedify/fedify stays available for compatibility. Contributed by ChanHaeng Lee (@2chanhaeng). (#418, #758)

Claude Code plugin

The Fedify repository now ships a Claude Code plugin at claude-plugin/, installable with:

/plugin marketplace add fedify-dev/fedify
/plugin install fedify@fedify

The plugin adds six slash commands (/fedify:fedify, /fedify:docs, /fedify:actor, /fedify:inbox, /fedify:migration, /fedify:fep) and two specialized agents (fedify-reviewer and fedify-debugger). The Agent Skills bundle introduced in 2.2.0 is the canonical skill source; claude-plugin/skills/fedify/ is the authoritative location, referenced from packages/fedify/skills/fedify/ via a symlink so both distribution paths stay in sync. (#489, #756)

Other changes

fedify init gains a --skip-install option that skips automatic dependency installation after scaffolding, useful for CI environments or monorepos that install from the root. Contributed by fru1tworld (@fru1tworld). (#720, #776)

New projects generated by fedify init for Node.js and Bun now use Oxfmt and Oxlint instead of Biome and ESLint. @fedify/lint gains a @fedify/lint/oxlint subpath export for Oxlint's JS plugin API, contributed by NyanRus (@nyanrus). (#702, #703, #760, #818)

@fedify/fixture gains createTestMeterProvider() and TestMetricRecorder for asserting OpenTelemetry metric measurements in runtime-agnostic tests. @fedify/testing's createFederation() mock now accepts a meterProvider option for the same purpose. (#316, #755)

@fedify/vocab adds the second-stage FEP-0837 economic-resource types: Agreement (an agreement reached between parties responding to a Proposal) and Commitment (a promised economic transaction referencing an Intent). Contributed by Samuel Brinkmann (@scammo). (#775, #817)

Explicit ActivityStreams Link objects in icon and image properties are now normalized to Image during decoding via the new exported normalizeLinkToImage() preprocessor. Blank node identifiers (_:b0) are no longer resolved against options.baseUrl in fromJsonLd() methods; they are left as null in the resulting instance's id field. (#790, #792)

Acknowledgments

Thanks to this release's contributors:

• ChanHaeng Lee (@2chanhaeng): @fedify/uri-template
• Jiwon Kwon (@sij411): @fedify/backfill
• Samuel Brinkmann (@scammo): FEP-0837 Agreement and Commitment vocabulary
• NyanRus (@nyanrus): @fedify/lint/oxlint Oxlint support
• fru1tworld (@fru1tworld): --skip-install option

See CHANGES.md for the complete changelog.