Benchmarking: `fedify bench` engine, scenario format, and JSON schema hosting

[dahlia]

Note

Sub-issue of #744. Before reading further, read #744 in full, including all of its comments, where the benchmarking tool's design is worked out in detail. This issue is one slice of that design and assumes the decisions recorded in those comments.

This is step 2 of 5. It depends on #782.

Scope

Stand up the fedify bench command in @fedify/cli with the core engine, the scenario format, and the published JSON Schema, landing runnable with the inbox scenario end-to-end against a benchmarkMode target. The rationale is in the #744 comments; the concrete decisions:

Scenario format

• YAML input; JSON is accepted because YAML is a superset. Results are emitted as JSON. The file is a suite: top-level version, target, defaults, actors, and scenarios, with an expect block per scenario.
• Templating is GitHub-Actions-style ${{ ... }}, kept logic-less (helper calls only). The $ prefix avoids the YAML gotcha where a value starting with { is parsed as a flow mapping. Generated payloads use typed directives (content: { generate: lorem, size: 2KB }) rather than string templates. Scalar-or-list coercion applies across the format (recipient, seed, collection, type, and so on).
• The tool owns actor URLs and activity ids, so the unique-id-per-request idempotency requirement is automatic rather than something an author can forget.

actors, keyed by signature standard

• An author picks signature standards, not key algorithms, and the key set is derived (a Fedify actor is inherently multi-key). The standards and their algorithms: draft-cavage-http-signatures-12 and rfc9421 are HTTP request signatures (RSA; rfc9421 signing is RSA only today); ld-signatures is a document signature (RsaSignature2017, RSA); fep8b32 is a document integrity proof (eddsa-jcs-2022, Ed25519).
• Validation: at most one HTTP request scheme per actor (draft-cavage-http-signatures-12 and rfc9421 are mutually exclusive), document schemes optional. A heterogeneous fleet (several actor groups with different standard sets) is supported.

Engine

• A single result model with renderers for terminal text, JSON, and Markdown (Markdown suited to a GitHub Actions job summary). JSON is the canonical machine form pinned by the schema; numbers carry their unit in the key name (latencyMs), client and server numbers are split, expect results are evaluated records, and environment/configHash are first-class.
• Load generation, open-loop and closed-loop, with coordinated-omission correction (latency measured from the scheduled time). Arrival distribution constant (default) or poisson; a maxInFlight saturation cap; HdrHistogram-style percentiles.
• A signing pipeline reusing the @fedify/fedify signer, with three lookahead modes: jit (sign in the send path, for strict-time-window black-box targets), pipeline (default; a bounded buffer kept filled by background signers, a unique activity id per buffered request, buffer starvation surfacing the client as the bottleneck), and presign (the whole run signed before the timed window, for rates beyond real-time signing throughput).
• The benchmark's own synthetic actor/key server, serving the actor and key documents the target dereferences during verification (a fixed actor set keeps this on a cold path that a warm-up window excludes).

JSON schema, single source of truth, and hosting

• Schema-first definitions (TypeBox or Zod) produce the TypeScript types, the runtime validator, and the published JSON Schema (draft 2020-12). Cross-field rules live in the schema: exactly one HTTP request scheme via contains + minContains/maxContains, rate XOR concurrency via oneOf, per-type fields and allowed expect metrics via if/then + propertyNames. Friendlier messages come from errorMessage annotations or a thin message-only post-pass; the schema stays authoritative.
• A schema/ directory in the monorepo, deployed by CI/CD to json-schema.fedify.dev. Files are append-only and versioned (a new version is a new file, never an edit), each $id equals its hosted URL, served with CORS and immutable caching. CI guards: meta-schema validation, an immutability guard against changing a published file, example-fixture validation, and a drift check against the schema-first source. Editor support via a # yaml-language-server: $schema= line.

Dependencies

Depends on #782: the tool targets a benchmarkMode app and reads its stats.

Acceptance criteria

• fedify bench runs the inbox scenario against a local benchmarkMode target and reports latency, throughput, success rate, and errors.
• YAML and JSON scenario files parse and validate against the schema; the cross-field rules (one HTTP scheme, rate XOR concurrency, per-type expect metrics) are enforced.
• JSON, Markdown, and terminal output from one result model, with client/server separation and unit-suffixed numeric keys.
• Open-loop and closed-loop both available, with coordinated-omission correction and HdrHistogram percentiles.
• pipeline signing is the default; jit and presign are selectable; all reuse the @fedify/fedify signer.
• The JSON Schema is published to json-schema.fedify.dev by CI/CD with the append-only, immutability, fixture, and drift guards.
• The editor $schema line is documented for scenario files.
• Tests for argument parsing, result aggregation, and the inbox scenario against a local test server.

Documentation

Create docs/manual/benchmarking.md and register it in the VitePress sidebar (docs/.vitepress/config.mts); cover the scenario format, the ${{ }} templating and actors model, and a getting-started example.