Skip to content

security(rate-limit): webhook bucket keyed on unverified installation.id — 10 unauthenticated req/min silences a target installation's webhooks #9469

Description

@JSONbored

Summary

The inbound rate-limit bucket for /v1/github/webhook and /v1/orb/webhook is keyed on installation.id read from the unverified request body, before signature verification, with no IP component. Any unauthenticated party can therefore pin a chosen installation's webhook bucket at its limit by sending 10 junk requests per minute, causing GitHub's genuine deliveries to be rejected with 429. GitHub does not automatically redeliver failed deliveries, so the engine goes deaf for that installation.

Cost to the attacker: one request every six seconds. No credentials required.

Mechanism (verified at HEAD, 776c414)

src/auth/rate-limit.ts:225-230:

const INSTALLATION_KEYED_WEBHOOK_PATHS = new Set(["/v1/github/webhook", "/v1/orb/webhook"]);
...
if (INSTALLATION_KEYED_WEBHOOK_PATHS.has(path)) {
  const installationId = await peekWebhookInstallationId(c);
  return installationId === null ? null : `installation:${installationId}`;
}

src/auth/rate-limit.ts:256-271 reads the id straight out of the body and says so:

// Reads installation.id from the body BEFORE signature verification ... The value is therefore
// unverified at this point -- fine for bucketing (a spoofed installation.id here only shares that
// installation's own rate-limit bucket; it grants no access, since HMAC/enrollment verification
// still gates everything downstream).

The threat note is correct that spoofing grants no access. It misses that sharing the bucket is the attack. The limit is strict = 10 requests / 60 seconds (src/auth/rate-limit.ts:24), and both webhook paths map to it (:138, :141).

Installation ids are low-entropy sequential integers and are discoverable (they appear in App installation URLs and in any webhook payload the target's own integrations expose).

Attack

  1. Attacker sends POST /v1/github/webhook with body {"installation":{"id":<victim>}} — no valid signature needed, because the bucket is charged before verification.
  2. 10 such requests in 60 s exhaust the victim installation's strict bucket.
  3. GitHub's real deliveries for that installation now receive 429 from the middleware (src/api/routes.ts:1199-1204, which runs ahead of handleGitHubWebhook).
  4. GitHub does not auto-redeliver. Those events are lost; the engine only recovers what its ~2-minute sweep happens to re-poll, at the cost of REST budget.
  5. Sustained at 10 req/min, the outage is indefinite.

Compounding: the limit is also undersized for legitimate traffic

#4891 (2026-07-18) collapsed what had been per-IP sharding into a single per-installation bucket, and the 10/min budget was never re-sized. GitHub delivers from many egress IPs, so the effective capacity used to be several × 10/min.

Evidence the comment is now stale — src/auth/rate-limit.ts:140 still reads:

the per-IP strict cap is proven for /v1/github/webhook

The App subscribes to check_run, check_suite, push, pull_request and more (src/github/webhook.ts:27-37). A single push to a repo with ~8 checks produces roughly 20 deliveries within a couple of minutes, counted before any noise filtering. Two concurrent PRs under one installation can plausibly exceed 10/min in normal operation — dropping real deliveries, which then self-heal only via sweep re-fetches that themselves consume REST budget.

Requirements

  1. An unauthenticated caller must not be able to consume another installation's webhook budget.
  2. Legitimate GitHub delivery bursts (CI storms) must not be throttled.
  3. Pre-authentication requests must still be rate-limited — the fix must not remove protection, only re-key it.

Deliverables

  • Re-key the webhook buckets so a spoofed body cannot collide with genuine traffic. Preferred: composite key installation:{id}:ip:{hash} so GitHub's egress IPs and an attacker's never share a bucket. Alternative: keep pure-IP keying pre-signature and apply the installation-keyed strict check after HMAC verification. Record which was chosen and why.
  • Re-size the webhook path budget for CI-storm reality — a dedicated class (e.g. 60–120/min per installation) rather than the blanket strict 10/min. Keep strict for the auth/ingest surfaces where it is appropriate.
  • Update the stale comment at :140 and the threat note at :259-261, which currently document a security property the code no longer has.
  • Add a metric distinguishing "webhook 429 for an installation with no valid signature seen recently" so a future attempt is visible rather than blending into the known scraper noise.

Tests (must fail against current main)

  • Unsigned requests carrying a victim installation.id do not consume the bucket that genuine signed deliveries for that installation use.
  • A burst of ~20 signed deliveries for one installation within 2 minutes is fully accepted.
  • Pre-auth requests are still limited (no regression in blanket protection).
  • /v1/orb/webhook covered identically to /v1/github/webhook.

Expected outcome

Webhook intake cannot be denied to an installation by an unauthenticated third party, and normal CI bursts stop being silently dropped.

Note on disclosure

This is a availability/DoS vector against the maintainer's own deployment and any fleet operator enrolled via the broker. Filing as a normal maintainer-only issue on that basis; if the fleet is treated as multi-tenant for disclosure purposes, handle accordingly before the fix ships.

Metadata

Metadata

Assignees

Labels

gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions