Skip to content

Observability

Jump to bottom
Chris edited this page Aug 15, 2026 · 110 revisions

Enterprise Observability & Telemetry

Value Proposition Gain clear visibility into mission-critical AI operations with continuous real-time telemetry. Implement a robust dual-pillar observability architecture using observability platforms like Prometheus, Grafana, and Datadog for minimal-overhead metrics export and comprehensive diagnostic auditing to monitor, measure, and securely optimize agent actions at scale. Read the full value proposition.

Establish Visibility

Establish a comprehensive operational baseline to ensure peak reliability and transparent operations. Clearly distinguish between the two complementary pillars of the observability architecture:

  • MCP Resources: These are robust database-specific endpoints seamlessly exposed to the AI agent. Examples include mysql://schema, mysql://insights, mysql://performance, and mysql://metrics. They also include mysql://sysschema for comprehensive wait events and I/O summaries. They empower the AI to proactively evaluate database state, performance metrics, and schema definitions with comprehensive clarity. See Resources for more details.
  • System Telemetry: This refers to passive metrics and logs. The server exports these to external monitoring tools. Examples include Prometheus, Grafana, and Datadog. This page focuses on setting up and using System Telemetry.

Configure Server Logging

The server outputs structured internal logs based on the configured log level. Set this via the --log-level flag or by using the LOG_LEVEL environment variable.

Note

The LOG_LEVEL environment variable does not use the MCP_ prefix. This is also true for other system configuration variables like MYSQL_POOL_SIZE and PROXYSQL_HOST.

Supported levels:

  • debug: Verbose output, including payload serialization sizes and internal function traces.
  • info: (Default) Standard operational logs, startup events, and significant connection lifecycle events.
  • warn: Recoverable errors or deprecation warnings.
  • error: Critical failures and unhandled exceptions.

Tip

Forensic Audit Logging: Administrators may require complete forensic JSONL logging. This includes all queries, mutations, and Code Mode executions. If so, configure the Audit Subsystem. See Audit Trail for detailed setup instructions.

Telemetry Persistence with Local Storage

mysql-mcp persists telemetry and agent activities to a local SQLite SystemDb. This provides audit trails, token usage tracking, and AI efficiency metrics out-of-the-box.

Export Prometheus Metrics

The server exports Prometheus metrics via the /metrics endpoint.

Important

Transport Behavior: The /metrics endpoint is natively exposed via the MCP streamable HTTP transport (--transport http / NodeStreamableHTTPServerTransport).

1. Native Export (Single Instance)

To enable the native metrics endpoint, pass the --metrics-export prometheus flag or set MCP_METRICS_EXPORT=prometheus.

When running the MCP server directly, this enables native Prometheus metrics. The transport behavior dictates how this is exposed:

  • HTTP Transport: The /metrics endpoint is natively exposed via the primary HTTP server port.
  • Stdio Transport: Passing --metrics-export prometheus will automatically spin up a standalone background HTTP server on port 3000 strictly to expose these metrics.

2. Aggregated Export (Multi-Agent/Distributed)

When running multiple agents or distributed deployments, it is recommended to use the dedicated mysql-mcp-exporter container to aggregate telemetry and prevent port conflicts. Note: The metrics server and exporter share a single process. Both operate on port 3000. This prevents port contention. The container reads the live mcp-audit.jsonl written by the MCP server, and must be configured to output its own audit events to a separate file (e.g., --audit-log /var/log/mysql-mcp/exporter-audit.jsonl). The AUDIT_LOG_PATH environment variable must also be explicitly set in the container (/var/log/mysql-mcp/mcp-audit.jsonl) so it knows where to find the MCP server's live JSONL file.

Metrics exposed include:

  • mysql_mcp_tool_calls_total: Total number of MCP tool invocations.
  • mysql_mcp_tool_errors_total: Total number of tool errors.
  • mysql_mcp_tool_latency_ms_p50: Median tool execution latency.
  • mysql_mcp_pool_queries_total: Total number of connection pool queries.
  • mysql_mcp_resource_reads_total: Total number of resource reads.
  • mysql_mcp_uptime_seconds: Server uptime.

To verify the exporter is running correctly, use the following healthcheck command:

wget --spider -q http://127.0.0.1:3000/metrics

Visualize with Grafana

The project's source repository includes pre-configured Grafana dashboards for both metrics and logs.

  • Metrics: Visualizes Prometheus metrics exposing mysql-mcp tool usage, latencies, and resource reads.
  • Logs: The MySQL-MCP Logs (Loki) dashboard visualizes the structured mcp-audit.jsonl stream, aggregated natively by Grafana Alloy (replaces EOL Promtail).
  • Location: The JSON definitions for the dashboards are located in the examples/full-observability-ecosystem/config/grafana/dashboards/ directory.
  • Access: Run the full ecosystem via docker compose up -d from the examples/full-observability-ecosystem directory. Grafana is then available at http://localhost:3001 (login: admin / admin). The dashboards are pre-loaded.

Token & Tool Metrics

Grafana Core Dashboard

AI Efficiency

Grafana AI Efficiency Dashboard

MySQL-MCP Logs (Loki)

Grafana Logs Dashboard

Grafana Alloy Routing

When configuring Grafana Alloy to collect logs, enforce strict routing rules to ensure the audit trail is properly processed:

  • Route the mcp-audit.jsonl stream strictly to Loki.

OpenTelemetry Guidelines

When implementing OpenTelemetry (OTel), adhere to the following tracing and instrumentation guidelines:

  • Semantic Conventions: Use the standard gen_ai.* semantic conventions for all LLM and agentic tracing.
  • Trace Propagation: Ensure standard traceparent headers are propagated across all service boundaries.
  • OTLP Exporters: Configure OTLP exporters over gRPC (4317) or HTTP (4318) using standard environment variables (e.g., OTEL_TRACES_EXPORTER=otlp) for optimal performance.
  • Batch Processors: Always use batch span processors to aggregate telemetry data before export, reducing network overhead.

Enable Datadog Telemetry

For observability, the project's source repository integrates with Datadog. This includes:

  • Autodiscovery: The Datadog Agent automatically discovers and monitors all containers.
  • eBPF System Probe: Captures deep kernel-level network performance metrics.
  • APM Tracing: Enabled for application containers to trace requests across boundaries.
  • Live Processes: Tracks host and container processes.
  • Custom Dashboards: Pre-configured dashboards include datadog-tool-performance.json for Token and Tool Metrics (including the live MySQL-MCP Audit Log stream), datadog-ai-efficiency.json for AI Efficiency, datadog-mysql.json for MySQL Cluster Telemetry, datadog-redis.json for Redis Telemetry, lib-agent-exec.json for Agent Execution Telemetry, datadog-infrastructure.json for Infrastructure Telemetry, and datadog-logs.json for Logs. These dashboard JSON files reside in the examples/dashboards/ directory.

Note

Audit Log Configuration: The Audit Log widget dynamically queries source:mysql_mcp log_type:mcp_audit to capture detailed agent activity. By default, mysql-mcp intelligently omits read scope tool payloads (like mysql_read_query) to protect sensitive enterprise data and optimize bandwidth. To actively stream read queries into Datadog, simply append the --audit-reads argument to the server's args array in the MCP client configuration file.

Note

MySQL Shell Status Widget: The MySQL Shell Status widget specifies .fill(last) behavior to ensure continuous visibility of the shell's operational state between discrete execution events.

Note

WSL2 UDP Routing: To route UDP telemetry across the Windows-WSL boundary, use wsl -c "nc -u" to bridge the UDP packets.

Datadog Configuration Hardening

To ensure a resilient and accurate Datadog deployment, apply the following configuration hardening measures:

  • CLI Authentication: Use pup auth to securely authenticate the Datadog CLI without exposing raw API keys.
  • WSL2 Resource Limits: In WSL2 environments, enforce strict container limits (e.g., mem_limit: 1536m, stop_grace_period: 30s, timeout: 10s for healthchecks).
  • OpenMetrics Integrations: Enable OpenMetrics scraping for Prometheus endpoints to seamlessly ingest custom telemetry.
  • Inotify Log Tailing: Configure the agent to use inotify for efficient file tailing of the audit log.
  • Duplicate Autodiscovery Warning: Ensure that multiple Docker integration endpoints are not configured simultaneously to prevent duplicate autodiscovery metrics.

Token & Tool Metrics

Datadog Core Dashboard

AI Efficiency

Datadog AI Efficiency Dashboard

MySQL Cluster Telemetry

Datadog MySQL Telemetry Dashboard

Redis Telemetry

Datadog Redis Telemetry Dashboard

Infrastructure Templates

Production-ready Docker Compose templates are provided in the examples/ directory of the repository. These templates include full Datadog observability out-of-the-box:

  • Basic Template: A lightweight, single-node MySQL setup with Datadog Agent.
  • Enterprise HA Template: A highly-available InnoDB Cluster with MySQL Router, ProxySQL, and Datadog Agent.
  • Full Observability Ecosystem: The complete open-source observability stack (Prometheus, Grafana, Loki, Alloy, Dozzle, Redis) along with Datadog integrations and the enterprise MySQL architecture.

See the README files in those directories for quick start instructions.

Stack Resilience

All observability containers (including Prometheus, Grafana Alloy, Exporter, and Datadog) are configured with restart: unless-stopped to gracefully survive host laptop reboots and Docker daemon restarts without manual intervention.

Configuration: End-users should start with the templates in the examples/ directory. The internal development test server configures the datadog-unified agent via docker-compose.yml in the test-server/infrastructure directory.

Monitor Logs with Dozzle

Streamline container log analysis and easily view and search container logs in real-time using Dozzle, included in the ecosystem. This can be done without the CLI.

  • Access: When running the test infrastructure, Dozzle is available at http://localhost:8080.

Explore Related Topics

MySQL MCP Documentation

Unlock autonomous database orchestration with an enterprise-grade MySQL MCP server. Featuring blazing-fast sandboxed Code Mode, uncompromising schema enforcement, and seamless ecosystem integrations to power secure, intelligent AI workflows.

🏠 Home

Launch Your Setup

Connect Ecosystem Tools

Enforce Security & Compliance

Scale Your Operations

Explore External Links

Clone this wiki locally