Skip to content

Troubleshooting

Jump to bottom
alonsalasi edited this page Aug 23, 2026 · 3 revisions

Troubleshooting

Pod Not Starting

Symptom: Pod exits immediately or stays in CrashLoopBackOff.

kubectl describe pod -n monitoring -l app.kubernetes.io/name=k8s-telemetry-mcp
kubectl logs -n monitoring -l app.kubernetes.io/name=k8s-telemetry-mcp

Common causes:

Log message Fix
MCP_AWS_MARKETPLACE_PRODUCT_CODE is not set Set config.marketplaceProductCode in Helm values
License check failed Verify product code is correct and EKS node role has both aws-marketplace:RegisterUsage and marketplace-entitlement:GetEntitlements permissions
ImagePullBackOff Re-create the ECR pull secret — the token may have expired

IAM permissions required on EKS node role:

{
  "Effect": "Allow",
  "Action": [
    "aws-marketplace:RegisterUsage",
    "marketplace-entitlement:GetEntitlements"
  ],
  "Resource": "*"
}

RegisterUsage is called every hour for billing. GetEntitlements is called once at startup to verify your active subscription tier. Both must be present or the pod will fail to start.

Cannot Connect to Loki / Prometheus / Tempo

# Check service URLs
kubectl get svc -n monitoring

# Test connectivity from inside the pod
kubectl exec -it -n monitoring deploy/k8s-telemetry-mcp -- \
  curl -s http://loki:3100/ready

Check that MCP_LOKI_URL, MCP_PROMETHEUS_URL, and MCP_TEMPO_URL match the actual service names and ports in your cluster.

Logs Returning Empty Results

query_pod_logs and analyze_logs return empty results when Loki has no data.

Loki does not collect logs on its own — you need a log shipper pushing logs into it.

# Check if a shipper is running
kubectl get pods -n monitoring | findstr /i "promtail fluent"

If nothing is returned, install Promtail:

helm upgrade --install promtail grafana/promtail \
  --namespace monitoring \
  --set config.lokiAddress=http://loki:3100/loki/api/v1/push

Then verify Loki has data:

curl "http://loki:3100/loki/api/v1/query?query={namespace=\"default\"}"

SLO Status Shows no_data

check_slo_status returns no_data when Prometheus has no http_requests_total metrics for the service.

Your application needs to emit Prometheus metrics. Options:

  • Add a Prometheus client library to your service
  • If using Istio, enable metrics collection
  • Verify Prometheus has a scrape config for your service: 
    kubectl get servicemonitor -n <your-namespace>

AI Assistant Not Discovering Tools

  1. Verify the pod is running: kubectl get pods -n monitoring
  2. Test the stdio connection manually: 
    echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | \
  kubectl exec -i -n monitoring deploy/k8s-telemetry-mcp -- k8s-telemetry-mcp
  3. Confirm the user running the AI assistant has kubectl exec permission on the deployment.

Tier Limit Exceeded Error

If you see Tier limit exceeded, your subscription tier restricts access. Standard allows 1 namespace and 8 core tools. Either:

  • Upgrade to Professional or Enterprise at https://aws.amazon.com/marketplace
  • If you have already upgraded, verify your EKS node role has marketplace-entitlement:GetEntitlements permission so the server can read your current entitlement. Without it, the server falls back to the MCP_MARKETPLACE_TIER env var.

Clone this wiki locally