Skip to content

Troubleshooting

Jump to bottom
alonsalasi edited this page Aug 23, 2026 · 3 revisions

Troubleshooting

Pod Not Starting

Symptom: Pod exits immediately or stays in CrashLoopBackOff.

kubectl describe pod -n monitoring -l app.kubernetes.io/name=k8s-telemetry-mcp
kubectl logs -n monitoring -l app.kubernetes.io/name=k8s-telemetry-mcp

Common causes:

Log message Fix
MCP_AWS_MARKETPLACE_PRODUCT_CODE is not set Set config.marketplaceProductCode in Helm values
License check failed Verify product code is correct and EKS node role has both aws-marketplace:RegisterUsage and marketplace-entitlement:GetEntitlements permissions
ImagePullBackOff Re-create the ECR pull secret — the token may have expired

IAM permissions required on EKS node role:

{
  "Effect": "Allow",
  "Action": [
    "aws-marketplace:RegisterUsage",
    "marketplace-entitlement:GetEntitlements"
  ],
  "Resource": "*"
}

RegisterUsage is called every hour for billing. GetEntitlements is called once at startup to verify your active subscription tier. Both must be present or the pod will fail to start.

Cannot Connect to Loki / Prometheus / Tempo

# Check service URLs
kubectl get svc -n monitoring

# Test connectivity from inside the pod
kubectl exec -it -n monitoring deploy/k8s-telemetry-mcp -- \
  curl -s http://loki:3100/ready

Check that MCP_LOKI_URL, MCP_PROMETHEUS_URL, and MCP_TEMPO_URL match the actual service names and ports in your cluster.

Logs Returning Empty Results

query_pod_logs and analyze_logs return empty results when Loki has no data.

Loki does not collect logs on its own — you need a log shipper pushing logs into it.

# Check if a shipper is running
kubectl get pods -n monitoring | findstr /i "promtail fluent"

If nothing is returned, install Promtail:

helm upgrade --install promtail grafana/promtail \
  --namespace monitoring \
  --set config.lokiAddress=http://loki:3100/loki/api/v1/push

Then verify Loki has data:

curl "http://loki:3100/loki/api/v1/query?query={namespace=\"default\"}"

SLO Status Shows no_data

check_slo_status returns no_data when Prometheus has no http_requests_total metrics for the service.

Your application needs to emit Prometheus metrics. Options:

  • Add a Prometheus client library to your service
  • If using Istio, enable metrics collection
  • Verify Prometheus has a scrape config for your service: 
    kubectl get servicemonitor -n <your-namespace>

AI Assistant Not Discovering Tools

  1. Verify the pod is running: kubectl get pods -n monitoring
  2. Test the stdio connection manually: 
    echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | \
  kubectl exec -i -n monitoring deploy/k8s-telemetry-mcp -- k8s-telemetry-mcp
  3. Confirm the user running the AI assistant has kubectl exec permission on the deployment.

Kubernetes API Tools Not Working

get_k8s_events, get_scaling_history, get_node_pressure, and get_recent_deployments require the Helm chart RBAC to be enabled.

# Verify the ClusterRole exists
kubectl get clusterrole k8s-telemetry-mcp

# If missing, upgrade with RBAC enabled
helm upgrade k8s-telemetry-mcp k8s-telemetry-mcp/k8s-telemetry-mcp \
  --namespace monitoring \
  --set rbac.enabled=true

If you see "Ensure the service account has 'get,list' on 'events'" in the tool response, the ClusterRole is missing or the ClusterRoleBinding is not pointing at the correct service account.

kube-state-metrics Not Installed

get_scaling_history returns a note field warning when no HPAs are found. For historical HPA metrics in Prometheus, install kube-state-metrics:

helm install kube-state-metrics prometheus-community/kube-state-metrics \
  --namespace monitoring

RDS Performance Insights Not Enabled

get_database_insights falls back to basic CloudWatch metrics when Performance Insights is disabled, and includes a warning field explaining this. To enable richer slow query data:

  1. Go to RDS Console → your instance → Modify
  2. Enable Performance Insights
  3. Re-run get_database_insights — it will automatically use the richer data source

ECR Inspector v2 Not Enabled

get_image_vulnerabilities falls back to ECR basic scan results when Inspector v2 is not active, and includes a warning field. To enable Inspector v2:

aws inspector2 enable --resource-types ECR

get_alertmanager_history Returns Error

Set MCP_ALERTMANAGER_URL in your Helm values:

config:
  alertmanagerUrl: "http://alertmanager.monitoring:9093"

If you see Tier limit exceeded, your subscription tier restricts access. Standard allows 1 namespace and 8 core tools. Either:

  • Upgrade to Professional or Enterprise at https://aws.amazon.com/marketplace
  • If you have already upgraded, verify your EKS node role has marketplace-entitlement:GetEntitlements permission so the server can read your current entitlement. Without it, the server falls back to the MCP_MARKETPLACE_TIER env var.

Clone this wiki locally