You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[Security] actor_mailbox_dropped_total carries a dynamic path label that under sharding derives from a remote-supplied entity id, so a remote party mints permanent, never-evicted metric series #745
Component:src/metrics/Metrics.ts Severity (assessment): LOW CWE: CWE-770 (Allocation of Resources Without Limits or Throttling) Related:#131
childOf creates a new child on first sight of a label tuple and the family's children map has no cap, no TTL and no eviction; clear() is the only removal path and is described as a test hook. actor_mailbox_dropped_total is the one stock family with dynamic labels, and its path label is the full actor path — which under cluster sharding is entity-<sanitised entityId>, i.e. a value chosen by whoever addresses the shard region.
Exploit walkthrough
Preconditions: metrics enabled (explicitly, or implicitly for the duration of a DevTools attach), cluster sharding in use, and a path by which an attacker influences entity ids — a cluster client, an HTTP handler that routes shardRegion.tell({ entityId: req.params.id, ... }), or a broker consumer.
Attacker addresses N distinct entity ids. Each spawns an entity actor at a distinct path.
For a given entity, the attacker sustains enough inbound volume to overflow that entity's mailbox (default DEFAULT_MAILBOX_CAPACITY = 10_000, policy drop-head) — which is the point of the counter.
Each overflowing entity contributes one permanent child series {class, path, reason}. The series is never removed, not even when the entity passivates and its actor is gone, because childOf has no eviction and nothing calls clear().
Cost to the defender is two-fold: heap held by the child map for the life of the process, and scrape size — exportPrometheus walks every child on every scrape (PrometheusExporter.ts:21-74) and the resulting exposition is what the Prometheus server must ingest and index, so a cardinality blow-up here lands on the monitoring system as well.
Honest cost accounting: the 10 000-message-per-entity overflow requirement makes this expensive per unit of cardinality, and it needs sustained backpressure rather than a burst — which is why this is LOW and not higher. It is nonetheless the one place where a remote party's choice of identifier becomes a permanent, unbounded allocation in the metrics registry.
Evidence — src/metrics/Metrics.ts
src/metrics/Metrics.ts:319-336 — unbounded child map, no eviction:
The design decision is stated up front (Metrics.ts:19-22: "Cardinality discipline is the user's responsibility … The registry doesn't enforce limits"), and the framework mostly honours it: actor_created_total, actor_terminated_total, actor_restarted_total, actor_messages_delivered_total, actor_message_handler_seconds, cluster_gossip_rounds_total and cluster_members_up all pass {} — Cluster.ts:945 even comments that the gauge's labels are "deliberately empty". entityName sanitises the id to [A-Za-z0-9_-], which stops exposition injection but not distinctness. NoopMetricsRegistry means a system that never enables metrics pays nothing. What is absent is any ceiling once metrics are on: no maxSeriesPerFamily, no LRU over children, no TTL, and no drop-with-warning when a family exceeds a threshold — so the one labelled family the framework itself ships has no backstop.
Suggested fix
Add an optional per-family series cap to DefaultMetricsRegistry (maxSeriesPerFamily, defaulting to something like 10 000) that, once exceeded, folds further label tuples into a single {overflow="true"} series and logs once — the standard client-library behaviour. Separately, reconsider whether path belongs on actor_mailbox_dropped_total at all: class plus reason already answers the operational question, and the exact path is available on the DeadLetter / event stream where it costs nothing per-series.
Relationship to existing issues
Adjacent to #131, but a distinct mechanism. Verified. src/metrics/Metrics.ts:319-336 childOf inserts into family.children with no cap, TTL or eviction (only clear() removes). src/internal/ActorCell.ts:827-834 confirms the counter is emitted with { class: cls, path: this.path.toString(), reason }, and src/cluster/sharding/Shard.ts:188-190 confirms the sharded child name is entity-${entityId.replace(/[^A-Za-z0-9_\-]/g, '_')} — user/remote-chosen. #131 already owns the generic mechanism ('DefaultMetricsRegistry creates a new child series on every distinct label combo', 'no per-family cardinality cap') and its fix (maxSeriesPerFamily + overflow series) would also fix this, which is why this is not DISTINCT. But #131 rests on an explicit factual claim that this finding refutes: 'The framework itself doesn't make this mistake in its built-in metrics (they all use static label sets like {} or {queue: mailbox} — verified by grep)'. That grep was wrong, and the consequence is different in kind: the risk in #131 is user code choosing a bad label, here it is the framework's own instrumentation carrying a remote-derived value, plus a second remediation #131 never considers (drop path from the stock family; class + reason already answer the operational question). Cross-reference #131 and correct its premise. LOW, and I would not go higher: minting one series costs the attacker ~10 000 messages to overflow that entity's mailbox, so the cardinality-per-effort ratio is poor.
Verification status
Found in the second, independent whole-framework security re-audit of 2026-08-02 (v0.12.0) — a fresh pass run without reference to the first wave's findings, then triaged against the existing tracker and adjudicated by verifiers instructed to refute it.
Verifier note
Code claims confirmed.
src/metrics/Metrics.ts:319-328 — childOf inserts into family.children unconditionally; there is no cap, TTL or eviction, and clear() (line 283) is the only removal path.
src/internal/ActorCell.ts:827-834 — the counter is emitted with { class: cls, path: this.path.toString(), reason }, and src/internal/ActorCell.ts:154-165 shows the bounded mailbox is the default (DEFAULT_MAILBOX_CAPACITY = 10_000, DEFAULT_MAILBOX_OVERFLOW = 'drop-head', src/util/Constants.ts:84,96), so the callback is wired for every actor.
src/cluster/sharding/Shard.ts:188-190 — entityName is entity-${entityId.replace(/[^A-Za-z0-9_\-]/g, '_')}, and Shard.ts:165 spawns the child under that name, so the path label really does carry a caller-chosen identifier.
Grepped every .counter( / .gauge( / .histogram( call site in src/: actor_created_total, actor_terminated_total, actor_restarted_total, actor_messages_delivered_total, actor_message_handler_seconds, cluster_gossip_rounds_total, cluster_members_up all pass {}. actor_mailbox_dropped_total is the sole stock family with dynamic labels — so [Security] Prometheus cardinality attack via user-controlled label values #131's "verified by grep" premise is wrong, and src/devtools/internal/MetricsDigest.ts:15-17 even documents the labelled shape in passing.
ActorPath.toString() does not include the uid (src/ActorPath.ts:136-141), so a respawned entity reuses its series rather than minting a new one — the growth is per distinct entity id, not per incarnation. Series do survive passivation, as claimed.
DevTools-induced accumulation is bounded by the attach: NodeSampler.stop() (src/devtools/internal/NodeSampler.ts:77-80) calls disable(), which replaces the registry (MetricsExtension.ts:66-68) and drops the accumulated children.
LOW confirmed; I would not raise it. The overflow precondition (sustained inbound above the entity's drain rate) makes the cardinality-per-effort ratio poor, and #131's cap already covers the mechanism.
Correction applied: Two corrections. (1) The resource-growth mechanism and its remediation are already #131's Track 1 (maxSeriesPerFamily + overflow series) — that cap fixes this too. What is genuinely new is the factual correction to #131 (its body claims the built-in metrics "all use static label sets like {} or {queue: 'mailbox'} — verified by grep", which is false) and the second remediation (drop path from the stock family). File this as a correction/cross-reference on #131, not as an independent DoS claim. (2) The exploit cost is understated as "10 000 messages": drop-head only fires while the mailbox is at capacity (src/mailbox/BoundedMailbox.ts:41-53), so the attacker must sustain an inbound rate above that entity's drain rate, not merely send 10 000 messages. Against a fast handler that is impractical; it needs a slow (e.g. persistence-backed) entity.
Component:
src/metrics/Metrics.tsSeverity (assessment): LOW
CWE: CWE-770 (Allocation of Resources Without Limits or Throttling)
Related: #131
childOfcreates a new child on first sight of a label tuple and the family'schildrenmap has no cap, no TTL and no eviction;clear()is the only removal path and is described as a test hook.actor_mailbox_dropped_totalis the one stock family with dynamic labels, and itspathlabel is the full actor path — which under cluster sharding isentity-<sanitised entityId>, i.e. a value chosen by whoever addresses the shard region.Exploit walkthrough
Preconditions: metrics enabled (explicitly, or implicitly for the duration of a DevTools attach), cluster sharding in use, and a path by which an attacker influences entity ids — a cluster client, an HTTP handler that routes
shardRegion.tell({ entityId: req.params.id, ... }), or a broker consumer.DEFAULT_MAILBOX_CAPACITY = 10_000, policydrop-head) — which is the point of the counter.{class, path, reason}. The series is never removed, not even when the entity passivates and its actor is gone, becausechildOfhas no eviction and nothing callsclear().exportPrometheuswalks every child on every scrape (PrometheusExporter.ts:21-74) and the resulting exposition is what the Prometheus server must ingest and index, so a cardinality blow-up here lands on the monitoring system as well.Honest cost accounting: the 10 000-message-per-entity overflow requirement makes this expensive per unit of cardinality, and it needs sustained backpressure rather than a burst — which is why this is LOW and not higher. It is nonetheless the one place where a remote party's choice of identifier becomes a permanent, unbounded allocation in the metrics registry.
Evidence —
src/metrics/Metrics.tssrc/metrics/Metrics.ts:319-336 — unbounded child map, no eviction:
src/internal/ActorCell.ts:827-834 — the labelled stock counter:
src/cluster/sharding/Shard.ts:188-190 — the path segment for a sharded entity is remote-derived:
src/devtools/internal/NodeSampler.ts:53-56 — DevTools silently promotes the noop registry to a real, accumulating one:
Why the existing guard does not cover it
The design decision is stated up front (Metrics.ts:19-22: "Cardinality discipline is the user's responsibility … The registry doesn't enforce limits"), and the framework mostly honours it:
actor_created_total,actor_terminated_total,actor_restarted_total,actor_messages_delivered_total,actor_message_handler_seconds,cluster_gossip_rounds_totalandcluster_members_upall pass{}—Cluster.ts:945even comments that the gauge's labels are "deliberately empty".entityNamesanitises the id to[A-Za-z0-9_-], which stops exposition injection but not distinctness.NoopMetricsRegistrymeans a system that never enables metrics pays nothing. What is absent is any ceiling once metrics are on: nomaxSeriesPerFamily, no LRU over children, no TTL, and no drop-with-warning when a family exceeds a threshold — so the one labelled family the framework itself ships has no backstop.Suggested fix
Add an optional per-family series cap to
DefaultMetricsRegistry(maxSeriesPerFamily, defaulting to something like 10 000) that, once exceeded, folds further label tuples into a single{overflow="true"}series and logs once — the standard client-library behaviour. Separately, reconsider whetherpathbelongs onactor_mailbox_dropped_totalat all:classplusreasonalready answers the operational question, and the exact path is available on theDeadLetter/ event stream where it costs nothing per-series.Relationship to existing issues
Adjacent to #131, but a distinct mechanism. Verified. src/metrics/Metrics.ts:319-336
childOfinserts intofamily.childrenwith no cap, TTL or eviction (onlyclear()removes). src/internal/ActorCell.ts:827-834 confirms the counter is emitted with{ class: cls, path: this.path.toString(), reason }, and src/cluster/sharding/Shard.ts:188-190 confirms the sharded child name isentity-${entityId.replace(/[^A-Za-z0-9_\-]/g, '_')}— user/remote-chosen. #131 already owns the generic mechanism ('DefaultMetricsRegistry creates a new child series on every distinct label combo', 'no per-family cardinality cap') and its fix (maxSeriesPerFamily+ overflow series) would also fix this, which is why this is not DISTINCT. But #131 rests on an explicit factual claim that this finding refutes: 'The framework itself doesn't make this mistake in its built-in metrics (they all use static label sets like {} or {queue: mailbox} — verified by grep)'. That grep was wrong, and the consequence is different in kind: the risk in #131 is user code choosing a bad label, here it is the framework's own instrumentation carrying a remote-derived value, plus a second remediation #131 never considers (droppathfrom the stock family;class+reasonalready answer the operational question). Cross-reference #131 and correct its premise. LOW, and I would not go higher: minting one series costs the attacker ~10 000 messages to overflow that entity's mailbox, so the cardinality-per-effort ratio is poor.Verification status
Found in the second, independent whole-framework security re-audit of 2026-08-02 (
v0.12.0) — a fresh pass run without reference to the first wave's findings, then triaged against the existing tracker and adjudicated by verifiers instructed to refute it.Verifier note
Code claims confirmed.
src/metrics/Metrics.ts:319-328—childOfinserts intofamily.childrenunconditionally; there is no cap, TTL or eviction, andclear()(line 283) is the only removal path.src/internal/ActorCell.ts:827-834— the counter is emitted with{ class: cls, path: this.path.toString(), reason }, andsrc/internal/ActorCell.ts:154-165shows the bounded mailbox is the default (DEFAULT_MAILBOX_CAPACITY = 10_000,DEFAULT_MAILBOX_OVERFLOW = 'drop-head',src/util/Constants.ts:84,96), so the callback is wired for every actor.src/cluster/sharding/Shard.ts:188-190—entityNameisentity-${entityId.replace(/[^A-Za-z0-9_\-]/g, '_')}, andShard.ts:165spawns the child under that name, so thepathlabel really does carry a caller-chosen identifier..counter(/.gauge(/.histogram(call site insrc/:actor_created_total,actor_terminated_total,actor_restarted_total,actor_messages_delivered_total,actor_message_handler_seconds,cluster_gossip_rounds_total,cluster_members_upall pass{}.actor_mailbox_dropped_totalis the sole stock family with dynamic labels — so [Security] Prometheus cardinality attack via user-controlled label values #131's "verified by grep" premise is wrong, andsrc/devtools/internal/MetricsDigest.ts:15-17even documents the labelled shape in passing.ActorPath.toString()does not include the uid (src/ActorPath.ts:136-141), so a respawned entity reuses its series rather than minting a new one — the growth is per distinct entity id, not per incarnation. Series do survive passivation, as claimed.NodeSampler.stop()(src/devtools/internal/NodeSampler.ts:77-80) callsdisable(), which replaces the registry (MetricsExtension.ts:66-68) and drops the accumulated children.LOW confirmed; I would not raise it. The overflow precondition (sustained inbound above the entity's drain rate) makes the cardinality-per-effort ratio poor, and #131's cap already covers the mechanism.
Correction applied: Two corrections. (1) The resource-growth mechanism and its remediation are already #131's Track 1 (
maxSeriesPerFamily+ overflow series) — that cap fixes this too. What is genuinely new is the factual correction to #131 (its body claims the built-in metrics "all use static label sets like{}or{queue: 'mailbox'}— verified by grep", which is false) and the second remediation (droppathfrom the stock family). File this as a correction/cross-reference on #131, not as an independent DoS claim. (2) The exploit cost is understated as "10 000 messages":drop-headonly fires while the mailbox is at capacity (src/mailbox/BoundedMailbox.ts:41-53), so the attacker must sustain an inbound rate above that entity's drain rate, not merely send 10 000 messages. Against a fast handler that is impractical; it needs a slow (e.g. persistence-backed) entity.