Skip to content

[Security] actor_mailbox_dropped_total carries a dynamic path label that under sharding derives from a remote-supplied entity id, so a remote party mints permanent, never-evicted metric series #745

Description

@pathosDev

Component: src/metrics/Metrics.ts
Severity (assessment): LOW
CWE: CWE-770 (Allocation of Resources Without Limits or Throttling)
Related: #131

childOf creates a new child on first sight of a label tuple and the family's children map has no cap, no TTL and no eviction; clear() is the only removal path and is described as a test hook. actor_mailbox_dropped_total is the one stock family with dynamic labels, and its path label is the full actor path — which under cluster sharding is entity-<sanitised entityId>, i.e. a value chosen by whoever addresses the shard region.

Exploit walkthrough

Preconditions: metrics enabled (explicitly, or implicitly for the duration of a DevTools attach), cluster sharding in use, and a path by which an attacker influences entity ids — a cluster client, an HTTP handler that routes shardRegion.tell({ entityId: req.params.id, ... }), or a broker consumer.

  1. Attacker addresses N distinct entity ids. Each spawns an entity actor at a distinct path.
  2. For a given entity, the attacker sustains enough inbound volume to overflow that entity's mailbox (default DEFAULT_MAILBOX_CAPACITY = 10_000, policy drop-head) — which is the point of the counter.
  3. Each overflowing entity contributes one permanent child series {class, path, reason}. The series is never removed, not even when the entity passivates and its actor is gone, because childOf has no eviction and nothing calls clear().
  4. Cost to the defender is two-fold: heap held by the child map for the life of the process, and scrape size — exportPrometheus walks every child on every scrape (PrometheusExporter.ts:21-74) and the resulting exposition is what the Prometheus server must ingest and index, so a cardinality blow-up here lands on the monitoring system as well.

Honest cost accounting: the 10 000-message-per-entity overflow requirement makes this expensive per unit of cardinality, and it needs sustained backpressure rather than a burst — which is why this is LOW and not higher. It is nonetheless the one place where a remote party's choice of identifier becomes a permanent, unbounded allocation in the metrics registry.

Evidence — src/metrics/Metrics.ts

src/metrics/Metrics.ts:319-336 — unbounded child map, no eviction:

private childOf<M>(family: Family, labels: Labels, factory: () => M): M {
  const key = labelKey(labels);
  const existing = family.children.get(key);
  if (existing) return existing.metric as unknown as M;
  const metric = factory();
  family.children.set(key, { labels: { ...labels }, metric: metric as never });
  return metric;
}

src/internal/ActorCell.ts:827-834 — the labelled stock counter:

private _onMailboxDrop(reason: 'drop-head' | 'drop-new'): void {
  const cls = this.actor?.constructor.name ?? 'unknown';
  metricsOf(this.system).counter(
    'actor_mailbox_dropped_total',
    { class: cls, path: this.path.toString(), reason },
    { help: 'Cumulative count of user messages dropped by a bounded mailbox\'s overflow policy.' },
  ).inc();
}

src/cluster/sharding/Shard.ts:188-190 — the path segment for a sharded entity is remote-derived:

export function entityName(entityId: string): string {
  return `entity-${entityId.replace(/[^A-Za-z0-9_\-]/g, '_')}`;
}

src/devtools/internal/NodeSampler.ts:53-56 — DevTools silently promotes the noop registry to a real, accumulating one:

const metrics = this.system.extension(MetricsExtensionId);
if (!metrics.isEnabled()) { metrics.enable(); this.enabledMetrics = true; }

Why the existing guard does not cover it

The design decision is stated up front (Metrics.ts:19-22: "Cardinality discipline is the user's responsibility … The registry doesn't enforce limits"), and the framework mostly honours it: actor_created_total, actor_terminated_total, actor_restarted_total, actor_messages_delivered_total, actor_message_handler_seconds, cluster_gossip_rounds_total and cluster_members_up all pass {}Cluster.ts:945 even comments that the gauge's labels are "deliberately empty". entityName sanitises the id to [A-Za-z0-9_-], which stops exposition injection but not distinctness. NoopMetricsRegistry means a system that never enables metrics pays nothing. What is absent is any ceiling once metrics are on: no maxSeriesPerFamily, no LRU over children, no TTL, and no drop-with-warning when a family exceeds a threshold — so the one labelled family the framework itself ships has no backstop.

Suggested fix

Add an optional per-family series cap to DefaultMetricsRegistry (maxSeriesPerFamily, defaulting to something like 10 000) that, once exceeded, folds further label tuples into a single {overflow="true"} series and logs once — the standard client-library behaviour. Separately, reconsider whether path belongs on actor_mailbox_dropped_total at all: class plus reason already answers the operational question, and the exact path is available on the DeadLetter / event stream where it costs nothing per-series.

Relationship to existing issues

Adjacent to #131, but a distinct mechanism. Verified. src/metrics/Metrics.ts:319-336 childOf inserts into family.children with no cap, TTL or eviction (only clear() removes). src/internal/ActorCell.ts:827-834 confirms the counter is emitted with { class: cls, path: this.path.toString(), reason }, and src/cluster/sharding/Shard.ts:188-190 confirms the sharded child name is entity-${entityId.replace(/[^A-Za-z0-9_\-]/g, '_')} — user/remote-chosen. #131 already owns the generic mechanism ('DefaultMetricsRegistry creates a new child series on every distinct label combo', 'no per-family cardinality cap') and its fix (maxSeriesPerFamily + overflow series) would also fix this, which is why this is not DISTINCT. But #131 rests on an explicit factual claim that this finding refutes: 'The framework itself doesn't make this mistake in its built-in metrics (they all use static label sets like {} or {queue: mailbox} — verified by grep)'. That grep was wrong, and the consequence is different in kind: the risk in #131 is user code choosing a bad label, here it is the framework's own instrumentation carrying a remote-derived value, plus a second remediation #131 never considers (drop path from the stock family; class + reason already answer the operational question). Cross-reference #131 and correct its premise. LOW, and I would not go higher: minting one series costs the attacker ~10 000 messages to overflow that entity's mailbox, so the cardinality-per-effort ratio is poor.

Verification status

Found in the second, independent whole-framework security re-audit of 2026-08-02 (v0.12.0) — a fresh pass run without reference to the first wave's findings, then triaged against the existing tracker and adjudicated by verifiers instructed to refute it.

Verifier note

Code claims confirmed.

  • src/metrics/Metrics.ts:319-328childOf inserts into family.children unconditionally; there is no cap, TTL or eviction, and clear() (line 283) is the only removal path.
  • src/internal/ActorCell.ts:827-834 — the counter is emitted with { class: cls, path: this.path.toString(), reason }, and src/internal/ActorCell.ts:154-165 shows the bounded mailbox is the default (DEFAULT_MAILBOX_CAPACITY = 10_000, DEFAULT_MAILBOX_OVERFLOW = 'drop-head', src/util/Constants.ts:84,96), so the callback is wired for every actor.
  • src/cluster/sharding/Shard.ts:188-190entityName is entity-${entityId.replace(/[^A-Za-z0-9_\-]/g, '_')}, and Shard.ts:165 spawns the child under that name, so the path label really does carry a caller-chosen identifier.
  • Grepped every .counter( / .gauge( / .histogram( call site in src/: actor_created_total, actor_terminated_total, actor_restarted_total, actor_messages_delivered_total, actor_message_handler_seconds, cluster_gossip_rounds_total, cluster_members_up all pass {}. actor_mailbox_dropped_total is the sole stock family with dynamic labels — so [Security] Prometheus cardinality attack via user-controlled label values #131's "verified by grep" premise is wrong, and src/devtools/internal/MetricsDigest.ts:15-17 even documents the labelled shape in passing.
  • ActorPath.toString() does not include the uid (src/ActorPath.ts:136-141), so a respawned entity reuses its series rather than minting a new one — the growth is per distinct entity id, not per incarnation. Series do survive passivation, as claimed.
  • DevTools-induced accumulation is bounded by the attach: NodeSampler.stop() (src/devtools/internal/NodeSampler.ts:77-80) calls disable(), which replaces the registry (MetricsExtension.ts:66-68) and drops the accumulated children.

LOW confirmed; I would not raise it. The overflow precondition (sustained inbound above the entity's drain rate) makes the cardinality-per-effort ratio poor, and #131's cap already covers the mechanism.

Correction applied: Two corrections. (1) The resource-growth mechanism and its remediation are already #131's Track 1 (maxSeriesPerFamily + overflow series) — that cap fixes this too. What is genuinely new is the factual correction to #131 (its body claims the built-in metrics "all use static label sets like {} or {queue: 'mailbox'} — verified by grep", which is false) and the second remediation (drop path from the stock family). File this as a correction/cross-reference on #131, not as an independent DoS claim. (2) The exploit cost is understated as "10 000 messages": drop-head only fires while the mailbox is at capacity (src/mailbox/BoundedMailbox.ts:41-53), so the attacker must sustain an inbound rate above that entity's drain rate, not merely send 10 000 messages. Against a fast handler that is impractical; it needs a slow (e.g. persistence-backed) entity.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: lowNice-to-have / niche / demand-drivenproduction-goalBlocks or defines the path to production readinesssecuritySecurity-relevant — see severity label for impact tierseverity: lowMinor / informational / mitigated-by-design

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions