Skip to content

[Docs] All four PromQL queries in the operations runbook name metrics that no code in src/ emits, and seven of the eight real stock metrics are never mentioned there, so every documented diagnostic returns an empty result #992

Description

@pathosDev

Where

  • docs/src/content/docs/operations/troubleshooting.mdx (all four PromQL blocks)
  • docs/src/content/docs/operations/tuning/dispatcher-tuning.mdx
  • docs/src/content/docs/operations/tuning/mailbox-sizing.mdx
  • docs/src/content/docs/operations/overview.mdx
  • docs/src/content/docs/operations/upgrades/rolling-migration.md
  • docs/src/content/docs/observability/management/http-endpoints.mdx (sample scrape output)
  • the German mirrors of all of the above under docs/src/content/docs/de/

What is wrong

The operations section is the runbook an on-call engineer opens at 3am. All four of its PromQL blocks query a metric family that src/ never creates. Against a real actor-ts process every one of them returns an empty result — which in Grafana, and in an alert rule, is indistinguishable from "the system is healthy".

The complete set of metric names emitted anywhere in src/ is eight. Every emission site, verbatim:

src/internal/ActorCell.ts:801-804
      metricsOf(this.system).counter(
        'actor_created_total', {},
        { help: 'Cumulative count of actors successfully started.' },
      ).inc();
src/internal/ActorCell.ts:889-892
    metricsOf(this.system).counter(
      'actor_terminated_total', {},
      { help: 'Cumulative count of actors that have been stopped.' },
    ).inc();
src/internal/ActorCell.ts:947-950
      metricsOf(this.system).counter(
        'actor_restarted_total', {},
        { help: 'Cumulative count of supervisor-driven actor restarts.' },
      ).inc();
src/internal/ActorCell.ts:967-971
    metricsOf(this.system).counter(
      'actor_mailbox_dropped_total',
      { class: cls, path: this.path.toString(), reason },
      { help: 'Cumulative count of user messages dropped by a bounded mailbox\'s overflow policy.' },
    ).inc();
src/internal/ActorCell.ts:987-990
    metrics.counter(
      'actor_messages_delivered_total', {},
      { help: 'Cumulative count of user messages delivered to actor onReceive.' },
    ).inc();
src/internal/ActorCell.ts:1044-1047
        metrics.histogram(
          'actor_message_handler_seconds', {},
          { help: 'Time spent inside actor onReceive handlers, seconds.' },
        ).observe(elapsedMs / 1000);
src/cluster/Cluster.ts:771-774
    metricsOf(this.system).counter(
      'cluster_gossip_rounds_total', {},
      { help: 'Cumulative count of gossip-push rounds initiated by this node.' },
    ).inc();
src/cluster/Cluster.ts:1102-1105
    metricsOf(this.system).gauge(
      'cluster_members_up', {},
      { help: 'Number of cluster members currently in `up` state.' },
    ).set(this.upMembers().length);

That is the whole set — actor_created_total, actor_terminated_total, actor_restarted_total, actor_mailbox_dropped_total, actor_messages_delivered_total, actor_message_handler_seconds, cluster_gossip_rounds_total, cluster_members_up. docs/src/content/docs/observability/metrics/stock-metrics.mdx lists exactly these eight and is correct; the operations section is the outlier.

Difference, docs → source. Six distinct names appear under docs/src/content/docs/operations/** that nothing in src/ emits.

docs/src/content/docs/operations/troubleshooting.mdx:69-72
```promql
rate(cluster_unreachable_duration_ms_count[5m]) > 0.1
# Frequent transitions
```
docs/src/content/docs/operations/troubleshooting.mdx:101-104
```promql
sharding_shards_hosted{type="entity-type"}
# Should equal numShards (total) across the cluster
```
docs/src/content/docs/operations/troubleshooting.mdx:148-150
```promql
histogram_quantile(0.99, persistence_recovery_duration_ms_bucket)
```
docs/src/content/docs/operations/troubleshooting.mdx:198-201
```promql
actor_mailbox_size{class=...}
# Find actors with persistently large queues
```
docs/src/content/docs/operations/tuning/dispatcher-tuning.mdx:141-155
Use the
[stock metric](/observability/metrics/stock-metrics/)
`actor_message_duration_ms`:

```
P50 ≈ work time
P99 - P50 ≈ dispatcher latency (queueing)
```

If P99 is far higher than P50 with little work-time variance,
**dispatcher tuning helps**.

The
`actor_mailbox_size` gauge under load shows whether actors are
keeping up.  Persistently growing depth = either a slow handler
docs/src/content/docs/operations/upgrades/rolling-migration.md:108-110
- v1 events are being read by the new code path
  (`migration_chain_upcast_total{from="1",to="2"}` metric, or a
  log-line spot-check on `chain.read`).

mailbox-sizing.mdx compounds it — it presents actor_mailbox_size as a stock metric and then names both metrics in the prose under a third spelling that matches neither the invented one nor the real one:

docs/src/content/docs/operations/tuning/mailbox-sizing.mdx:131-144
Stock metrics
([Stock metrics](/observability/metrics/stock-metrics/))
expose mailbox depth:

```
actor_mailbox_size{class="Worker", path="..."}
actor_mailbox_dropped_total{class="Worker", path="...", reason="drop-head"}
```

Watch:

- **`mailbox_size`** — high values relative to capacity indicate
  pressure.
- **`mailbox_dropped_total`** — non-zero with `drop-head` /

The section overview repeats the promise at the top:

docs/src/content/docs/operations/overview.mdx:129-132
<Aside type="caution" title="The second place is metrics">
  Stock metrics give you actor message-rate, mailbox depth, cluster
  member count, sharding region count — enough to identify
  "something is off" before users notice.

Neither mailbox depth nor sharding region count is emitted by anything in src/.

Difference, source → docs. Of the eight metrics the framework actually emits, seven never appear anywhere under docs/src/content/docs/operations/**. The one exception is actor_mailbox_dropped_total at mailbox-sizing.mdx:137. The runbook and the instrumentation share one name out of eight.

Two adjacent sites belong in the same sweep. docs/src/content/docs/observability/management/http-endpoints.mdx:138-140 shows a sample GET /metrics response for actor_messages_processed_total{class="Worker",path="..."} — wrong name (the real one is actor_messages_delivered_total) and labels that family does not carry. By contrast docs/src/content/docs/operations/tuning/failure-detector.mdx:185 uses cluster_unreachable_total legitimately: it is a user-written sample that registers the counter itself, not a framework claim.

Every error above is mirrored verbatim in docs/src/content/docs/de/operations/.

Fix

Rewrite the operations diagnostics against the eight metrics that exist, and stop advertising ones that do not.

  • troubleshooting.mdx — replace all four PromQL blocks. Unreachable flapping is visible as a drop in cluster_members_up plus the cluster event stream, not as a cluster_unreachable_duration_ms histogram. Mailbox pressure is rate(actor_mailbox_dropped_total[5m]), not a depth gauge. Sharding coverage and persistence-recovery latency have no metric at all — say so and point at the management endpoints and the logs, rather than at a query that silently returns nothing.
  • dispatcher-tuning.mdxactor_message_duration_ms is actor_message_handler_seconds (seconds, not milliseconds); delete the actor_mailbox_size paragraph.
  • mailbox-sizing.mdx — delete actor_mailbox_size; use the one real name consistently in both the code block and the bullets.
  • operations/overview.mdx — remove "mailbox depth" and "sharding region count" from the stock-metric promise.
  • rolling-migration.mdmigration_chain_upcast_total does not exist; the log-line spot-check it already offers as the alternative is the only real option.
  • observability/management/http-endpoints.mdx — fix the sample scrape to a real name and real labels.
  • Mirror every change into docs/src/content/docs/de/.

The invented metrics are individually reasonable things to want. Where one is genuinely wanted it belongs in a feature issue — mailbox depth is #196 — not in a runbook as though it shipped.

Add a guard so the drift cannot recur: a test that collects metric-name literals from src/ and asserts every metric-shaped identifier in docs/** is either in that set or inside a block explicitly marked as user-registered. tests/unit/config/NoDeadConfigKeys.test.ts already does exactly this for HOCON keys. #470 proposes the general doc-drift CI guard (typecheck fenced samples, assert documented defaults) — a metric-name check is the same mechanism applied to a different vocabulary and belongs in the same harness.

Acceptance sketch

  • Every PromQL query under docs/src/content/docs/operations/** names a metric that a grep of src/ finds as a string literal.
  • actor_mailbox_size, actor_message_duration_ms, sharding_shards_hosted, persistence_recovery_duration_ms, cluster_unreachable_duration_ms and migration_chain_upcast_total appear nowhere in docs/.
  • operations/overview.mdx promises only metrics that exist.
  • observability/management/http-endpoints.mdx shows a scrape sample with a real name and real labels.
  • The German mirror matches the English one line for line.
  • A test fails when a metric name appears in docs/ that no src/ file emits.

Verification status

Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: confirmed by reading. The emitted set was built by grepping every .counter( / .gauge( / .histogram( call site in src/ and reading the name argument at each — 8 sites, 8 names, all quoted above. The docs set was built with grep -rnoE "\b(actor|cluster|sharding|persistence|migration|mailbox)_[a-z0-9_]+" docs/src/content/docs/operations/. Both set differences are stated above and reproduce from those two greps alone.

Related: #196 asks for the mailbox-depth histogram these docs already describe as shipped; #658 and #745 concern the label set of the one stock metric the runbook does name correctly. #744 is the adjacent failure mode where GET /metrics answers 200 with an empty body — same 3am symptom of a query returning nothing, different cause. This issue is the docs-vs-code divergence only; it asks for no new metric.

Part of the production-readiness review batch — tracked in #913.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationpriority: highTop priority — high impact, plan nextproduction-goalBlocks or defines the path to production readiness

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions