Observability

Enterprise Observability & Telemetry

Value Proposition Gain clear visibility into mission-critical AI operations with continuous real-time telemetry. Implement a robust dual-pillar observability architecture using observability platforms like Prometheus, Grafana, and Datadog for minimal-overhead metrics export and comprehensive diagnostic auditing to monitor, measure, and securely optimize agent actions at scale. Read the full value proposition.

---

Establish Visibility

Establish a comprehensive operational baseline to ensure peak reliability and transparent operations. Clearly distinguish between the two complementary pillars of the observability architecture:

• MCP Resources: These are robust database-specific endpoints seamlessly exposed to the AI agent. Examples include mysql://schema, mysql://insights, mysql://performance, and mysql://metrics. They also include mysql://sysschema for comprehensive wait events and I/O summaries. They empower the AI to proactively evaluate database state, performance metrics, and schema definitions with comprehensive clarity. See Resources for more details.
• System Telemetry: This refers to passive metrics and logs. The server exports these to external monitoring tools. Examples include Prometheus, Grafana, and Datadog. This page focuses on setting up and using System Telemetry.

---

Configure Server Logging

The server outputs structured internal logs based on the configured log level. Set this via the --log-level flag or by using the LOG_LEVEL environment variable.

Note

The LOG_LEVEL environment variable does not use the MCP_ prefix. This is also true for other system configuration variables like MYSQL_POOL_SIZE and PROXYSQL_HOST.

Supported levels:

• debug: Verbose output, including payload serialization sizes and internal function traces.
• info: (Default) Standard operational logs, startup events, and significant connection lifecycle events.
• warn: Recoverable errors or deprecation warnings.
• error: Critical failures and unhandled exceptions.

Tip

Forensic Audit Logging: Administrators may require complete forensic JSONL logging. This includes all queries, mutations, and Code Mode executions. If so, configure the Audit Subsystem. See Audit Trail for detailed setup instructions.

---

Export Prometheus Metrics

The server exports Prometheus metrics via the /metrics endpoint.

Important

Transport Behavior: The /metrics endpoint is natively exposed via the MCP streamable HTTP transport (--transport http / NodeStreamableHTTPServerTransport).

1. Native Export (Single Instance)

To enable the native metrics endpoint, pass the --metrics-export prometheus flag or set MCP_METRICS_EXPORT=prometheus.

When running the MCP server directly, this enables native Prometheus metrics. The transport behavior dictates how this is exposed:

• HTTP Transport: The /metrics endpoint is natively exposed via the primary HTTP server port.
• Stdio Transport: Passing --metrics-export prometheus will automatically spin up a standalone background HTTP server on port 3000 strictly to expose these metrics.

2. Aggregated Export (Multi-Agent/Distributed)

When running multiple agents or distributed deployments, it is recommended to use the dedicated mysql-mcp-exporter container to aggregate telemetry and prevent port conflicts. Note: The metrics server and exporter share a single process. This consolidated architecture avoids port contention by multiplexing both services over a single port (3000). To prevent port 3000 contention, native metrics export (--metrics-export) must be explicitly disabled on the primary MCP server when running the standalone multiplexed exporter on the same host.

The container reads the live mcp-audit.jsonl written by the MCP server, and must be configured to output its own audit events to a separate file (e.g., --audit-log /var/log/mysql-mcp/exporter-audit.jsonl). The AUDIT_LOG_PATH environment variable must also be explicitly set in the container (/var/log/mysql-mcp/mcp-audit.jsonl) so it knows where to find the MCP server's live JSONL file. For Datadog ingestion, ensure the volume containing this file is mounted into the Datadog agent and configured via a custom conf.yaml or Docker Autodiscovery label to tail the JSONL file.

Metrics exposed include:

• mysql_mcp_tool_calls_total: Total number of MCP tool invocations (filterable by tool).
• mysql_mcp_tool_errors_total: Total number of tool errors (filterable by tool and error_type).
• mysql_mcp_tool_errors_by_category_total: Tool errors categorized by root cause.
• gen_ai_usage_prompt_tokens_total: Total tokens estimated across tool calls (GenAI semantic convention).
• gen_ai_usage_prompt_tokens_per_call: Derived gauge for average token overhead per call.
• mysql_mcp_tool_latency_ms_p50, _p95, _p99: Tool execution latency percentiles.
• mysql_mcp_pool_queries_total, mysql_mcp_pool_connections_total, mysql_mcp_pool_utilization_ratio: Connection pool telemetry.
• mysql_mcp_resource_reads_total: Total resource reads (filterable by resource URI).
• mysql_mcp_cache_hits_total, mysql_mcp_cache_misses_total: Metadata cache efficiency.
• mysql_mcp_redis_connected, mysql_mcp_redis_rate_limit_exceeded_total, mysql_mcp_redis_fallback_to_memory_total: Distributed rate limiter health.
• mysql_mcp_redis_lua_eval_latency_p95_ms: Redis execution telemetry.
• mysql_mcp_http_errors_total: HTTP transport layer error tracking.
• mysql_mcp_uptime_seconds: Server uptime.

To verify the exporter is running correctly, use the following healthcheck command:

curl -f http://127.0.0.1:3000/metrics || exit 1

---

Visualize with Grafana

Transform raw telemetry into actionable insights. The project's source repository includes pre-configured Grafana dashboards for both metrics and logs.

• Metrics: Visualizes Prometheus metrics exposing mysql-mcp tool usage, latencies, and resource reads.
• Logs: The MySQL-MCP Logs (Loki) dashboard visualizes the structured mcp-audit.jsonl stream, aggregated natively by Grafana Alloy.
• Location: The JSON definitions for the dashboards are located in the examples/full-observability-ecosystem/config/grafana/dashboards/ directory.
• Access: Run the full ecosystem via docker compose up -d from the examples/full-observability-ecosystem directory. Grafana is then available at http://localhost:3001 (login: admin / admin). The dashboards are pre-loaded.

Token & Tool Metrics

AI Efficiency

MySQL-MCP Logs (Loki)

Grafana Alloy Routing

When configuring Grafana Alloy to collect logs, enforce strict routing rules to ensure the audit trail is properly processed:

• Route the mcp-audit.jsonl stream strictly to Loki.

---

OpenTelemetry Guidelines

When implementing OpenTelemetry (OTel), adhere to the following tracing and instrumentation guidelines:

• Semantic Conventions: Use the standard gen_ai.* semantic conventions for all LLM and agentic tracing.
• Trace Propagation: Ensure standard traceparent and tracestate headers are propagated across all service boundaries.
• OTLP Exporters: Configure OTLP exporters over gRPC (4317) or HTTP (4318) using standard environment variables (e.g., OTEL_TRACES_EXPORTER=otlp) for optimal performance.
• Batch Processors: Always use batch span processors to aggregate telemetry data before export, reducing network overhead.

---

Enable Datadog Telemetry

For observability, the project's source repository integrates with Datadog. This includes:

• Autodiscovery: The Datadog Agent automatically discovers and monitors all containers.
• eBPF System Probe: Captures deep kernel-level network performance metrics.
• APM Tracing: Enabled for application containers to trace requests across boundaries.
• Live Processes: Tracks host and container processes.
• Custom Dashboards: Pre-configured dashboards include datadog-tool-performance.json for Token and Tool Metrics, datadog-ai-efficiency.json for AI Efficiency, datadog-mysql.json for MySQL Cluster Telemetry, datadog-redis.json for Redis Telemetry, datadog-agent-exec.json for Agent Execution Telemetry, datadog-infrastructure.json for Infrastructure Telemetry, and datadog-logs.json for Logs (including the live MySQL-MCP Audit Log stream). The Datadog dashboard JSON files reside in the examples/dashboards/ directory.

Note

Audit Log Configuration: The Audit Log widget dynamically queries source:mysql_mcp log_type:mcp_audit to capture detailed agent activity. By default, mysql-mcp intelligently omits read scope tool payloads (like mysql_read_query) to protect sensitive enterprise data and optimize bandwidth. To actively stream read queries into Datadog, simply append the --audit-reads argument to the server's args array in the MCP client configuration file.

Note

MySQL Shell Status Widget: The MySQL Shell Status widget specifies .fill(last) behavior to ensure continuous visibility of the shell's operational state between discrete execution events.

Note

WSL2 UDP Routing: To route UDP telemetry across the Windows-WSL boundary, use wsl -c "nc -u" to bridge the UDP packets.

Datadog Configuration Hardening

To ensure a resilient and accurate Datadog deployment, apply the following configuration hardening measures:

• CLI Authentication: Use standard API/APP keys or datadog-ci to securely authenticate Datadog without exposing raw keys in logs.
• WSL2 Resource Limits: In WSL2 environments, enforce strict container limits (e.g., mem_limit: 1536m, stop_grace_period: 30s, timeout: 10s for healthchecks).
• Performance Metrics: Set DD_EXTRA_PERFORMANCE_METRICS: false to avoid high ingestion overhead from deprecated system profiling metrics.
• OpenMetrics Integrations: Enable OpenMetrics scraping for Prometheus endpoints to seamlessly ingest custom telemetry.
• Inotify Log Tailing & WSL2 Warning: While inotify is recommended for native Linux environments, volume bind-mounts from Windows to WSL2 do not trigger inotify events over the 9P filesystem. In mixed host environments, run the server inside WSL2 or restart the Datadog Agent container (docker restart datadog-unified) if file modification events freeze.
• Duplicate Autodiscovery Anti-Pattern: NEVER combine static conf.yaml check files with Docker Autodiscovery labels (com.datadoghq.ad.*) for containerized services; reserve static conf.yaml strictly for host-level checks to prevent duplicate check runs.

Token & Tool Metrics

AI Efficiency

MySQL Cluster Telemetry

Redis Telemetry

Infrastructure Templates

Production-ready Docker Compose templates are provided in the examples/ directory of the repository. These templates include full Datadog observability out-of-the-box:

• Basic Template: A lightweight, single-node MySQL setup with Datadog Agent.
• Enterprise HA Template: A highly-available InnoDB Cluster with MySQL Router, ProxySQL, and Datadog Agent.
• Full Observability Ecosystem: The complete open-source observability stack (Prometheus, Grafana, Loki, Alloy, Dozzle, Redis) along with Datadog integrations and the enterprise MySQL architecture.

See the README files in those directories for quick start instructions.

Stack Resilience

All observability containers (including Prometheus, Grafana Alloy, Exporter, and Datadog) are configured with restart: unless-stopped to gracefully survive host laptop reboots and Docker daemon restarts without manual intervention.

---

Configuration: End-users should start with the templates in the examples/ directory. The internal development test server configures the datadog-unified agent via docker-compose.yml in the test-server/infrastructure directory.

Monitor Logs with Dozzle

Streamline container log analysis and easily view and search container logs in real-time using Dozzle, included in the ecosystem. This can be done without the CLI.

• Access: When running the test infrastructure, Dozzle is available at http://localhost:8080.

---

Explore Related Topics

• Tools
• Resources
• Audit Trail