[Security] Prometheus cardinality attack via user-controlled label values

[pathosDev]

Severity / Size

• Severity: MEDIUM
• Size: S
• Threat model: any deployment exposing Prometheus-style metrics where a label value is derived from attacker-controlled input (URL path, header value, body field, user-supplied ID). The threat is monitoring-system DoS — exhaust the metrics backend, not the actor system.

Affected files

• src/metrics/Metrics.ts:204-220 — MetricsRegistry.counter/gauge/histogram accept any Labels; no per-family cardinality cap.
• src/metrics/Metrics.ts:230-244 — DefaultMetricsRegistry creates a new child series on every distinct label combo.
• src/metrics/PromClientAdapter.ts:157-168 — adapter mirrors raw label values into the prom-client side; no normalisation.
• src/metrics/PromClientAdapter.ts (docs at line 19-22) — currently documents "Cardinality discipline is the user's job" but provides no helper.

Background

Prometheus's data model is one time-series per unique label-tuple. If a label can take 1 million distinct values, you get 1 million time-series. Each series carries cumulative storage cost (typically ~3–10 KiB resident at scrape time, more on disk). At enough cardinality:

• Prometheus's TSDB block builder OOMs during compaction.
• prom-client's in-process Registry grows linearly — memory exhaustion on the exposing node, not just the scrape target.
• register.metrics() text serialisation exceeds reasonable response sizes (a 100 MB scrape response chokes the scraper).

The user-controlled path is the easy one:

metrics.counter('http_requests_total', { path: req.path }).inc();

Each unique req.path value spawns a new series. An attacker hitting /api/x?id=1, /api/x?id=2, …, /api/x?id=1M produces 1M series.

The framework itself doesn't make this mistake in its built-in metrics (they all use static label sets like {} or {queue: 'mailbox'} — verified by grep). The risk is user code doing it. The framework can:

1. Provide a cardinality cap per metric family — refuse to mint new series past the cap, increment a single overflow series instead.
2. Provide a bucketize helper that maps high-cardinality values to a bounded bucket set ("known routes" + "other").
3. Document the failure mode prominently — the current docs mention it in passing in Metrics.ts:19-22, but the threat-model section in README.md doesn't flag it.

Exploit walkthrough

Step 1 — App author wires per-path metrics (common pattern from tutorials):

const httpReqs = metrics.counter('http_requests_total', { path: '<placeholder>' });
// in handler:
metrics.counter('http_requests_total', { path: req.path, method: req.method }).inc();

Step 2 — Attacker iterates URLs:

GET /a   GET /b   GET /aa   GET /ab   ...

A simple bash loop generates 1 million distinct paths in under a minute. Each one creates a new time-series in prom-client's in-process registry.

Step 3 — Prom-client's memory grows. Each Counter child carries a JS object with label hash, value, timestamps. At ~500 bytes per series in V8: 1M × 500B = 500 MB resident in the actor-ts process. Process OOMs or starts GCing aggressively.

Step 4 — If prom-client survives: scrape time. register.metrics() walks every series and emits text:

http_requests_total{path="/a",method="GET"} 1
http_requests_total{path="/b",method="GET"} 1
... × 1_000_000

Response is ~70 MB. Prometheus's scrape config caps response size (default 100 MB) — at 1M unique paths the scrape succeeds but ingestion fails: TSDB tries to allocate 1M new series at once and OOMs the Prometheus server.

Step 5 — Cascade: Prometheus down → Alertmanager fires nothing → real outages slip through → grafana dashboards empty → ops blind. Meanwhile attacker's 1M series sit in the in-process registry forever (prom-client doesn't expire) and the actor-ts process is permanently at +500MB RSS.

Realistic worst case: monitoring infrastructure outage, not framework outage. Recovery requires restarting the actor-ts process (clears in-process registry) + restarting Prometheus (clears WAL).

How the 8 already-landed security fixes inform this

• Wire-frame DoS cap — exact same shape: bound an unbounded growth vector at the boundary. Default cap, opt-in override, fail-closed when the cap is hit. Apply the same pattern to label-tuple cardinality per family.
• Gossip version cap — bound the version counter at 24h skew so a malicious peer can't claim arbitrary growth. Same shape: cap per-family series count at a sensible default (e.g., 1024); refuse new series past the cap.
• Snapshot seq integrity — validated that the input was within sane bounds before trusting it. Same shape: validate label-values for shape (length, character class) before minting a series.

The shape across all three: trust no input that can grow without bound — cap at a default, allow opt-in override.

Fix design

Track 1 — Per-family cardinality cap (primary). Default cap: 1024 distinct label-tuples per family. When the cap is hit, increment a single "...{__overflow__='1'}" series instead of minting a new child. Log a one-shot warning per family with the offending labels.

export interface MetricsRegistryOptions {
  /**
   * Maximum distinct label-tuples per family before overflow.  Default: 1024.
   * Set to Infinity to disable (NOT recommended for user-facing label values).
   */
  readonly maxSeriesPerFamily?: number;
}

class DefaultMetricsRegistry implements MetricsRegistry {
  private readonly maxSeries: number;
  private readonly overflowWarned: Set<string> = new Set();

  constructor(opts: MetricsRegistryOptions = {}) {
    this.maxSeries = opts.maxSeriesPerFamily ?? 1024;
  }

  private childOf<M>(family: Family, labels: Labels, factory: () => M): M {
    const key = labelsToKey(labels);
    const existing = family.children.get(key);
    if (existing) return existing.metric as M;
    if (family.children.size >= this.maxSeries) {
      if (!this.overflowWarned.has(name)) {
        this.overflowWarned.add(name);
        // log one-shot warning with sample of offending labels
        console.warn(
          `metrics: family "${name}" exceeded maxSeriesPerFamily=${this.maxSeries}; `
          + `additional series merged into __overflow__=1. Sample of offending labels: ${JSON.stringify(labels)}`,
        );
      }
      const overflowLabels = { __overflow__: '1' };
      const overflowKey = labelsToKey(overflowLabels);
      let entry = family.children.get(overflowKey);
      if (!entry) {
        entry = { labels: overflowLabels, metric: factory() as never };
        family.children.set(overflowKey, entry);
      }
      return entry.metric as M;
    }
    const metric = factory();
    family.children.set(key, { labels, metric: metric as never });
    return metric;
  }
}

Track 2 — bucketize helper for known patterns. Public API:

/**
 * Map a high-cardinality value to a bounded bucket set.  Returns `value`
 * if it matches one of `allowed`, otherwise `'other'`.
 */
export function bucketize<T extends string>(value: string, allowed: ReadonlyArray<T>): T | 'other' {
  return (allowed as readonly string[]).includes(value) ? (value as T) : 'other';
}

// usage:
const ALLOWED_ROUTES = ['/api/users/:id', '/api/posts/:id', '/health'] as const;
metrics.counter('http_requests_total', {
  path: bucketize(routeTemplate(req.path), ALLOWED_ROUTES),  // routeTemplate strips :id
}).inc();

Track 3 — Apply cap in the prom-client adapter. The adapter at PromClientAdapter.ts:170-189 (getOrCreateCounter etc.) doesn't have its own family-level series cache — it relies on prom-client's Counter.labels() to mint series. Apply the cap at the adapter layer too: maintain a Set<string> of seen label-keys per family, refuse new ones past the cap, merge into overflow.

Track 4 — Documentation update. README "Known security caveats" gets a bullet:

- Prometheus cardinality: the framework's metrics registry caps at 1024
  distinct label-tuples per family by default (`maxSeriesPerFamily`).
  Override at registry construction.  When a user-controlled value reaches
  a label, use the `bucketize` helper or pre-aggregate.

API surface

// MetricsRegistry construction:
new DefaultMetricsRegistry({ maxSeriesPerFamily: 4096 });

// Helper for user code:
import { bucketize } from 'actor-ts/metrics';

metrics.counter('http_requests_total', {
  path: bucketize(req.routeTemplate, ALLOWED_ROUTES),
  method: req.method,
}).inc();

// Adapter:
promClientRegistry({ client, registry, maxSeriesPerFamily: 1024 });

Backward compatibility

Non-breaking for the framework's own metrics (all use static label sets). Breaking for user code that minted >1024 distinct series per family — but that's the exact case being protected against. Document in CHANGELOG + provide opt-out via maxSeriesPerFamily: Infinity.

Test plan

1. Cap-respect test — mint 1025 distinct series, verify the 1025th lands in __overflow__=1 and a one-shot warning was emitted.
2. Cap-disabled test — maxSeriesPerFamily: Infinity lets the test mint 5000 series without overflow.
3. bucketize test — known values pass through, unknown values map to 'other'.
4. Overflow correctness — overflow series accumulates correctly across multiple inc calls.
5. Different families don't share cap — counter('foo') and counter('bar') each get their own 1024 budget.
6. Adapter parity — same cap applies through promClientRegistry; prom-client receives at most maxSeriesPerFamily + 1 distinct label-tuples per family.
7. Regression — collect() output for an overflowed family still parses correctly and includes the __overflow__ series.

Acceptance criteria

• MetricsRegistry honors maxSeriesPerFamily (default 1024).
• Overflow into __overflow__='1' series + one-shot warning per family.
• bucketize<T>(value, allowed) helper exported.
• promClientRegistry adapter applies the same cap.
• CHANGELOG entry + README "Known security caveats" updated.
• Test suite covers cap-respect, cap-disabled, bucketize, overflow correctness, family isolation.