Test Ecosystem

Test Ecosystem Setup Guide

Value Proposition Validate AI agents in an isolated test ecosystem. It replicates deployment topologies to verify boundaries and operations before production. Read the full value proposition.

Deploy and manage a comprehensive test ecosystem featuring InnoDB Cluster, MySQL Router, and ProxySQL. Securely validate AI agents in a pristine, isolated sandbox prior to production, ensuring high reliability and operational confidence for MySQL deployments. The stack is configured with Enterprise-grade Observability via Datadog integrations, running alongside Prometheus and Grafana, to provide metrics observability and APM. See Observability for more details. Proper container healthchecks and Unified Service Tagging (tags.datadoghq.com/env, tags.datadoghq.com/service, tags.datadoghq.com/version) are explicitly required for all node deployments.

Tip

Looking for end-user templates? This page documents the heavy internal E2E testing infrastructure. To deploy an optimized, end-user Docker template for isolated projects, utilize the examples/ directory in the main repository.

Note

Benchmarks: This test ecosystem includes comprehensive VM sandbox lifecycle benchmarks to validate sandbox pool performance, security validation, and context creation overhead.

---

1. 🚀 Quick Start: E2E Validation via Idempotent Deployments

Accelerate your validation pipeline with fully automated, idempotent deployments. Our automation scripts bootstrap Group Replication, reliably simulating production environments to orchestrate complex AI workflows.

cd test-server/infrastructure
node scripts/recreate-ecosystem.mjs

This master script will:

• Tear down the existing cluster and volumes (docker compose down -v).
• Start the fresh test containers (docker compose up -d).
• Automatically poll the nodes until they are healthy.
• Initialize the primary node as the cluster creator.
• Join secondary nodes to the cluster.
• Output the final cluster topology.

---

2. 🏛️ Production-Grade Realistic Topologies

This test ecosystem includes the core database architecture alongside a comprehensive observability stack to validate the mysql-mcp server:

┌─────────────────────────────────────────────────────────────────────────────┐
│               Docker Network: unified-database-ecosystem-net                │
│                                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐                       │
│  │ mysql-node1  │  │ mysql-node2  │  │ mysql-node3  │                       │
│  │   PRIMARY    │  │  SECONDARY   │  │  SECONDARY   │                       │
│  │  Cls: 3307   │  │  Cls: 3308   │  │  Cls: 3309   │                       │
│  │  X: 33061    │  │  X: 33062    │  │  X: 33063    │                       │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘                       │
│         │                 │                 │                                │
│         ├────────────┬────┴─────────────────┘                                │
│         ▼            ▼                                                       │
│ ┌───────────────────┐┌─────────────────────────┐    ┌──────────────┐  ┌──────────────┐ │
│ │mysql-async-replica││       MySQL Router      │    │   ProxySQL   │  │ redis-server │ │
│ │     Cls: 3310     ││ RW:6446    RO:6447/6448 │    │ Admin: 6032  │  │  Port: 6379  │ │
│ │     X: 33064      ││     REST API: 8443      │    │ Data:  6033  │  │              │ │
│ └───────────────────┘└─────────────────────────┴────┴──────────────┘  └──────────────┘ │
│                                                                              │
│            ┌──────────────────┐           ┌──────────────┐  ┌──────────────┐ │
│            │    Prometheus    │           │   Grafana    │  │   Adminer    │ │
│            │   Port: 9090     │           │  Port: 3001  │  │  Port: 8081  │ │
│            └──────────────────┘           └──────────────┘  └──────────────┘ │
│                                                                              │
│            ┌──────────────────┐           ┌──────────────┐                   │
│            │       Loki       │           │  Grafana     │                   │
│            │   Port: 3100     │           │  Alloy       │                   │
│            └──────────────────┘           │ Port: 12345  │                   │
│                                           └──────────────┘                   │
│            ┌──────────────────┐           ┌──────────────┐                   │
│            │      Dozzle      │           │   Datadog    │                   │
│            │    Port: 8080    │           │ Ports: 8125/ │                   │
│            └──────────────────┘           │   udp, 4318  │                   │
│                                           └──────────────┘                   │
│            ┌──────────────────┐           ┌────────────────────────┐         │
│            │mysql-mcp-exporter│           │ mysql-router-telemetry │         │
│            │    Port: 3000    │           │                        │         │
│            └──────────────────┘           └────────────────────────┘         │
└─────────────────────────────────────────────────────────────────────────────┘

---

3. 🚑 Disaster Recovery Operations

🔄 Reset Data (Re-seed E2E Test DB)

Rapidly restore a pristine state. You may need to re-seed the testdb for E2E validation. This avoids the overhead of tearing down the entire cluster:

cd test-server/infrastructure
node scripts/reset-database.mjs

🛠️ Automated Cluster Recovery

If containers are restarted and the cluster fails to auto-recover via SET PERSIST, you do not need to run manual scripts. The cluster-healer Docker sidecar constantly monitors the cluster topology. If it detects a complete outage, it automatically executes dba.rebootClusterFromCompleteOutage('mcpCluster') and STOP/START GROUP_REPLICATION via mysqlsh. To monitor recovery events:

docker logs -f cluster-healer

🛡️ Host Restart Resilience

All containers in the ecosystem are configured with restart: unless-stopped to gracefully survive host laptop reboots and Docker daemon restarts without manual intervention.

---

4. 🔌 Network Configurations

🐬 MySQL (via InnoDB Cluster)

Note

The local test cluster uses non-standard ports (3307-3310). This differs from the default 3306 shown in standard configuration examples. This avoids conflicts with local installations.

Property	Value
Primary (R/W)	tcp://localhost:3307 (Classic), tcp://localhost:33061 (X-Protocol)
Read Replicas (R/O)	tcp://localhost:3308, tcp://localhost:3309 (Classic), tcp://localhost:33062, tcp://localhost:33063 (X-Protocol)
Async Replica (R/O)	tcp://localhost:3310 (Classic), tcp://localhost:33064 (X-Protocol)
User	root
Password	root

🚦 MySQL Router

Property	Value
REST API URL	https://localhost:8443
REST API User	rest_api
REST API Password	router_api
RW Routing Port	6446 (Classic) (→ PRIMARY)
RO Routing Port	6447 (Classic), 6448 (X-Protocol) (→ SECONDARY)

📊 Observability & Tools

Property	URL
Datadog & OpenTelemetry	Ports: 8125/udp, 8126/tcp (APM), 4318 (OTLP HTTP). JSON Dashboards: Use your Datadog provisioning tool (e.g., pup, Terraform, or API) to sync AI Efficiency, Token & Tool Metrics, MySQL Cluster Telemetry, Redis Telemetry, and Agent Execution Telemetry. Semantic Conventions: Adhere to gen_ai.* semantic conventions and ensure traceparent propagation. OTLP Receiver: Configure OTLP exporters with batch processors. OpenMetrics Scraping: Ensure Prometheus endpoint scraping is enabled with timeout: 10s. WSL2 Resource Limits: Set mem_limit: 1536m to avoid instability. MySQL Node Limits: Ensure mysql-router and proxysql utilize stop_grace_period: 30s to prevent abrupt terminations during Datadog agent shutdowns. Datadog Configs: Ensure DD_EXTRA_PERFORMANCE_METRICS: false, pid: host (with default-cgroupns-mode: host), and DD_HOSTNAME (use adamic-wsl2 for WSL) are configured correctly. inotify Tailing: Configure log tailing targeting the physical file path (e.g., /var/log/mysql-mcp/mcp-audit.jsonl), explicitly warning against using the AUDIT_LOG_PATH environment variable for Datadog/OTel sidecars. Avoid Duplicate Integrations: Prevent duplicate Docker autodiscovery.
Prometheus	http://localhost:9090
mysql-mcp-exporter	http://localhost:3000 (Standalone metrics server. Healthcheck: wget --spider -q http://127.0.0.1:3000/metrics)
mysql-router-telemetry	(Background metrics scraper for MySQL Router)
Loki	http://localhost:3100
Grafana Alloy	http://localhost:12345 (internal HTTP status). Ensure strict routing rules (e.g., mcp-audit.jsonl strictly to Loki).
Grafana	http://localhost:3001 (admin/admin)
Dozzle (Logs)	http://localhost:8080
Adminer (DB UI)	http://localhost:8081 (Server: mysql-node1 (uses internal Docker network port 3306), User: root, Pass: root)
ProxySQL Admin	tcp://127.0.0.1:6032
ProxySQL Data	tcp://127.0.0.1:6033
Redis	tcp://localhost:6379

---

5. ⚠️ Windows Development: WSL2 Guidance

While this ecosystem can be deployed on any standard Docker host, Windows users running native docker-ce inside WSL2 Ubuntu (without Docker Desktop) may encounter a known WSL2 idle timeout behavior. The distro may terminate without active Windows-side client sessions, stopping all containers.

Windows-Specific Workarounds

🛡️ Maintain the Keepalive Mechanism

A Windows Scheduled Task (WSL-KeepAlive) runs at user logon. It is automatically registered by recreate-ecosystem.mjs. It launches wsl.exe -d <Distro-Name> --exec sleep infinity with a hidden window. This holds the distro alive indefinitely.

📡 UDP Telemetry Bridging

When running Datadog Agent or other UDP-based telemetry in a Windows-WSL environment, UDP packets must be explicitly routed across the Windows-WSL boundary. You can achieve this packet bridging using wsl -c "nc -u". In WSL2 environments, you must also enforce strict container resource limits for the Datadog Agent (mem_limit: 1536m).

🩺 Diagnose Crashes

If containers are cycling (green → red → green repeatedly):

1. Check if WSL is rebooting: wsl bash -c "journalctl --list-boots". Multiple short-lived boots indicates a WSL termination issue.
2. Identify the root cause: Run wsl bash -c "dmesg | grep InitTerminateInstanceInternal". This message means WSL sent systemctl poweroff to the distro.
3. Check the keepalive task: Get-ScheduledTask -TaskName 'WSL-KeepAlive' | Select State — must be Running.
4. Check Docker daemon: wsl bash -c "systemctl status docker" — must be active (running).
5. Check iptables backend: Certain host kernel environments require iptables-nft. If Docker fails to start, check /etc/docker/daemon.json and update-alternatives --display iptables.

📄 Review Key Config Files

File	Purpose
%USERPROFILE%\.wslconfig	WSL2 VM config: memory, swap, vmIdleTimeout=-1
/etc/docker/daemon.json	Docker storage driver, log rotation
/etc/systemd/system/wsl-keepalive.service	Backup in-distro keepalive (defense-in-depth)

---

Explore Related Topics

• MySQL Router - Connect and query through MySQL Router
• ProxySQL - ProxySQL integration
• MySQL Shell - Run MySQL Shell scripts
• Redis Integration - Redis rate limiting and test isolation
• Observability - Telemetry and monitoring setup