Native OpenTelemetry support — picking up the "unify metrics across instances" thread from #875

[DrFaust92]

Context

PR #875 (CLOUDP-373012: Support Operational Metrics via EventEmitter interface) was
closed in January with:

Closing as POC. We would want to explore more options particularly approaches that
unify metrics across instances.

This is an attempt to make the case that OpenTelemetry is a good answer to exactly that
open question, and to ask whether you'd be receptive to it.

The argument: scrape can't unify, push can

The current model is a Prometheus registry exposed on the monitoring server's /metrics
(src/transports/monitoringServer.ts). That's a pull model, and pull requires every
instance to be a stable, long-lived, network-addressable scrape target.

That holds for one deployment shape — HTTP transport, long-running, in an orchestrator.
It does not hold for the shape that's arguably more common for this server: stdio
transport, one short-lived process per client session, on a developer's machine or
inside an agent runtime. Those instances are unscrapable in principle. No amount of
registry work fixes it, because nothing can reach them before they exit.

OTLP push inverts that, and it's the same mechanism for both shapes:

• stdio/ephemeral instances push before exit; no scrape target needed
• long-running HTTP instances push on an interval, or keep being scraped via the
  Prometheus exporter — OTel's metrics SDK supports both readers concurrently
• everything lands in one backend, aggregated by resource attributes
  (service.instance.id, deployment.environment, …) rather than by scrape topology

That is "unify metrics across instances," and it's a solved problem in a spec with
existing Node SDKs rather than a bespoke EventEmitter contract you'd have to design,
document, version, and support.

Secondary benefit: traces, which metrics can't substitute for

Separate from the aggregation argument, tracing answers questions the current histogram
can't. mcp_tool_execution_duration_seconds{tool_name,status,error_type} tells you
which tool was slow. A trace tells you why — which is nearly always "which MongoDB
operation inside it was slow." Since the server already depends on the mongodb driver,
@opentelemetry/instrumentation-mongodb would give per-query spans essentially for free,
and MCP request IDs (cf. #1301, requestIdAttr) are a natural span-context carrier for
correlating a client's agent request through to the database operation.

Why this is cheaper for you than it looks

The seams already exist, largely from #1006 and #1079:

• TransportRunnerBase accepts an injected metrics (src/transports/base.ts:238 —
  metrics ?? new PrometheusMetrics(...))
• there's a Metrics interface (packages/metrics/src/types.ts) rather than a hard
  dependency on prom-client
• CreateMonitoringServerFn allows fully external monitoring-server management

So an OTel metrics bridge is close to additive: a second Metrics implementation
backed by the OTel API, selected by config. Traces are the larger piece of work, since
they need span creation at tool-execution and connection boundaries rather than just an
exporter swap.

What we'd suggest

1. Opt-in, default off — no behaviour change for existing users, no new egress by default.
2. Honour the standard OTEL_* environment variables
   (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_TRACES_EXPORTER, OTEL_SERVICE_NAME, …) rather
   than inventing MDB_MCP_* equivalents, so it composes with existing collector setups.
3. Keep it clearly distinct from MDB_MCP_TELEMETRY. That flag means "usage analytics to
   MongoDB," and this means "operational data to the operator's own backend." Conflating
   them in docs or config would be actively harmful — an operator disabling telemetry for
   privacy reasons must not also lose their own observability. Worth noting CLOUDP-373012: Support Operational Metrics via EventEmitter interface #875 hit a
   version of this problem: its rejected first approach routed connection events through
   the telemetry emitter, which the PR body itself flags as a privacy concern.
4. Optionally scope the first cut to metrics only (the bridge above) and defer traces,
   if that's an easier increment to land.

Our situation, for whatever it's worth as a data point

We run the server on Kubernetes with HTTP transport, --readOnly, against self-managed
MongoDB, scraped by Prometheus via the monitoring server, and we ship metrics to an
OTel-based pipeline. So we're the easy case — we can already scrape, and OTel traces
would be an improvement rather than a necessity for us. We're raising it because the
ephemeral-instance gap looks like the harder half of the problem you named in #875, and
because the fix generalises to both shapes.

Would you be open to a design discussion or a PR here? If there's already internal
MCP-project work on this, we're happy to just follow along instead.

---

Related but separate in scope: #1396, a much smaller gap where collectProcessMetrics is unreachable from the CLI/container. Filed on its own since it stands independently of the OTel question.