MW K8S Agent Debugging Cheat Sheet

MW-Agent Debugging & Troubleshooting Cheat Sheet

1. Quick Health Check

# Are all mw-agent pods running?
kubectl get pods -n mw-agent-ns

kubectl describe pod <MW_AGENT_POD> -n <NAMESPACE>

# Quick overview of everything in the namespace
kubectl get all -n mw-agent-ns

# Check events for errors
kubectl get events -n mw-agent-ns --sort-by='.lastTimestamp'

2. Verify Core Agent Components

These should always be running after installation.

DaemonSet — one pod per node for OS/kubelet metrics:

kubectl get daemonset mw-kube-agent -n mw-agent-ns

Deployment — cluster-wide telemetry:

kubectl get deployment mw-kube-agent -n mw-agent-ns

Config Updater:

kubectl get deployment mw-kube-agent-config-updater -n mw-agent-ns

Check if DaemonSet is running on all nodes:

# Compare DESIRED vs READY columns
kubectl get ds mw-kube-agent -n mw-agent-ns -o wide

3. Verify Auto-Instrumentation Components

Only present if mw-autoinstrumentation.enabled=true.

kubectl get deployment mw-agent-opentelemetry-operator -n mw-agent-ns
kubectl get deployment mw-auto-injector -n mw-agent-ns
kubectl get deployment mw-lang-aggregator -n mw-agent-ns
kubectl get daemonset mw-lang-detector -n mw-agent-ns

Check Instrumentation CR exists:

kubectl get otelinst mw-autoinstrumentation -n mw-agent-ns
kubectl get instrumentations.opentelemetry.io mw-autoinstrumentation -n mw-agent-ns

Inspect full Instrumentation CR:

kubectl get otelinst mw-autoinstrumentation -n mw-agent-ns -o yaml
kubectl get instrumentations.opentelemetry.io mw-autoinstrumentation -n mw-agent-ns -o yaml

4. Verify OpsAI

Only present if opsai.enabled=true.

kubectl get deployment opsai -n mw-agent-ns

5. Log Inspection

5.1 Core Agent Logs

# DaemonSet logs
kubectl logs -f daemonset/mw-kube-agent -n mw-agent-ns

# Deployment logs
kubectl logs -f deployment/mw-kube-agent -n mw-agent-ns

# Config updater logs
kubectl logs -f deployment/mw-kube-agent-config-updater -n mw-agent-ns

# Previous container logs (if crash-looping)
kubectl logs <POD_NAME> -n mw-agent-ns --previous

5.2 Auto-Instrumentation Logs

# Language detector
kubectl logs daemonset/mw-lang-detector -n mw-agent-ns

# Admission webhook / auto-injector
kubectl logs deploy/mw-auto-injector -n mw-agent-ns

# OpenTelemetry Operator
kubectl logs deploy/mw-agent-opentelemetry-operator -n mw-agent-ns

# Language aggregator
kubectl logs deploy/mw-lang-aggregator -n mw-agent-ns

5.3 Search for Errors

kubectl logs -n mw-agent-ns -l app.kubernetes.io/instance=mw-agent \
  --all-containers | grep -i "error\|fatal\|panic"

6. Network and Connectivity

The MW agent service exposes two OTEL endpoints:

gRPC — mw-service.mw-agent-ns:9319
HTTP — mw-service.mw-agent-ns:9320

Verify the service exists:

kubectl get svc mw-service -n mw-agent-ns

Test connectivity from an app pod to the agent:

# HTTP endpoint
kubectl exec -it <APP_POD> -n <APP_NAMESPACE> -- \
  curl http://mw-service.mw-agent-ns:9320

# gRPC endpoint
kubectl exec -it <APP_POD> -n <APP_NAMESPACE> -- \
  curl http://mw-service.mw-agent-ns:9319

DNS resolution test:

kubectl exec -it <APP_POD> -n <APP_NAMESPACE> -- \
  nslookup mw-service.mw-agent-ns

Outbound connectivity (Agent to Middleware Backend):

kubectl exec -it <MW_AGENT_POD> -n mw-agent-ns -- \
  curl -v https://<MW_TARGET_URL>

7. Helm Debugging

# Check current release status
helm status mw-agent -n mw-agent-ns

# View deployed values
helm get values mw-agent -n mw-agent-ns

# View ALL values including defaults
helm get values mw-agent -n mw-agent-ns --all

# View release history
helm history mw-agent -n mw-agent-ns

# Dry-run an upgrade to preview changes
helm upgrade mw-agent middleware-labs/mw-kube-agent-v3 \
  --dry-run --debug \
  -f values.yaml \
  -n mw-agent-ns

# Force restart config updater to pick up new config
kubectl rollout restart deployment \
  mw-kube-agent-config-updater -n mw-agent-ns

8. Auto-Instrumentation Debugging

8.1 Apps Not Detected

# Check lang-detector logs
kubectl logs daemonset/mw-lang-detector -n mw-agent-ns

# Restart lang-detector if detection is stuck
kubectl rollout restart daemonset/mw-lang-detector -n mw-agent-ns

# Verify app pods are Running
kubectl get pods -n <APP_NAMESPACE> -l app=<APP_LABEL>

Checklist:

App uses a supported language? (Java, Node.js, Python, .NET, Go, Apache HTTPD, Nginx)
Main app container is listed first in spec.containers?
App pod is in Running state?

8.2 Instrumentation Not Working

# Check auto-injector webhook logs
kubectl logs deploy/mw-auto-injector -n mw-agent-ns

# Check OTel operator logs
kubectl logs deploy/mw-agent-opentelemetry-operator -n mw-agent-ns

# Verify annotations on your pod template
kubectl get deployment <APP_DEPLOYMENT> -n <APP_NAMESPACE> \
  -o jsonpath='{.spec.template.metadata.annotations}' | jq .

# Restart pods after enabling instrumentation
kubectl rollout restart deployment/<APP_DEPLOYMENT> -n <APP_NAMESPACE>

Checklist:

Namespace is in auto-instrumentation scope?
Pods restarted after enabling?
Annotations under spec.template.metadata.annotations? (NOT top-level metadata.annotations)
Network connectivity from app pod to mw-service.mw-agent-ns:9319 / 9320?

8.3 Manual Annotation Quick Reference

Java:

instrumentation.opentelemetry.io/inject-java: "mw-agent-ns/mw-autoinstrumentation"

Node.js:

instrumentation.opentelemetry.io/inject-nodejs: "mw-agent-ns/mw-autoinstrumentation"

Python:

instrumentation.opentelemetry.io/inject-python: "mw-agent-ns/mw-autoinstrumentation"

Python on Alpine (musl):

instrumentation.opentelemetry.io/inject-python: "mw-agent-ns/mw-autoinstrumentation"
instrumentation.opentelemetry.io/otel-python-platform: "musl"

.NET (glibc):

instrumentation.opentelemetry.io/inject-dotnet: "mw-agent-ns/mw-autoinstrumentation"
instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-x64"

.NET (musl):

instrumentation.opentelemetry.io/inject-dotnet: "mw-agent-ns/mw-autoinstrumentation"
instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-musl-x64"

Go (requires privileged security context):

instrumentation.opentelemetry.io/inject-go: "mw-agent-ns/mw-autoinstrumentation"
instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/path/to/executable"

Apache HTTPD:

instrumentation.opentelemetry.io/inject-apache-httpd: "mw-agent-ns/mw-autoinstrumentation"

Nginx:

instrumentation.opentelemetry.io/inject-nginx: "mw-agent-ns/mw-autoinstrumentation"

8.4 Multi-Container Pods

Single language — specify which containers:

instrumentation.opentelemetry.io/inject-java: "mw-agent-ns/mw-autoinstrumentation"
instrumentation.opentelemetry.io/java-container-names: "myapp,myapp2"

Multiple languages — use language-specific container-names:

instrumentation.opentelemetry.io/inject-java: "mw-agent-ns/mw-autoinstrumentation"
instrumentation.opentelemetry.io/java-container-names: "myapp"
instrumentation.opentelemetry.io/inject-python: "mw-agent-ns/mw-autoinstrumentation"
instrumentation.opentelemetry.io/python-container-names: "myapp2"

Important notes:

Go does NOT support multi-container pods
A single container cannot be instrumented with multiple languages
If container-names is not set, the first container gets instrumented (could be a sidecar!)

8.5 Namespace Scoping

# Include only specific namespaces
--set mw-autoinstrumentation.webhook.includedNamespaces="{app,backend}"

# Exclude specific namespaces
--set mw-autoinstrumentation.webhook.userExcludedNamespaces="{monitoring,infra}"

# Skip app detection for specific namespaces
--set mw-autoinstrumentation.langDetector.userExcludedNamespaces="{app,backend}"

9. Common Issues and Fixes

No data in Middleware dashboard

Cause: Agent not running or no outbound internet access
Fix: Check pod status, verify outbound connectivity to MW target URL

mw-kube-agent OOMKilled or crash-looping

Cause: Default resource limits too low for large clusters
Fix: Increase resources.limits.memory and resources.limits.cpu in values.yaml

DaemonSet not scheduled on all nodes

Cause: Node taints that pods cannot tolerate (common with Karpenter or custom pools)
Fix: Add appropriate tolerations in values.yaml

Multiple agents running

Cause: More than one MW agent DaemonSet deployed in the cluster
Fix: Run only ONE mw-agent DaemonSet per cluster

Apps not detected

Cause: Unsupported language, or sidecar container is listed first
Fix: Verify supported language, reorder containers so app is first

Instrumentation not injecting

Cause: Pods not restarted after enabling, or annotation typo/wrong path
Fix: Restart pods, verify annotations are under spec.template.metadata.annotations

Webhook or cert errors

Cause: cert-manager not installed, or TLS certs expired
Fix: Install cert-manager, or use self-signed certs (see Section 12)

OTel CRD conflict on install

Cause: Existing OpenTelemetry Operator CRDs in cluster
Fix: Delete old CRDs or skip CRD creation (see Section 11)

Config changes not taking effect

Cause: Config updater has not picked up the new values
Fix:

kubectl rollout restart deployment mw-kube-agent-config-updater -n mw-agent-ns

10. Provider-Specific Troubleshooting

GKE Private Cluster — Webhook Port 9443 Blocked

Find the control-plane CIDR:

gcloud container clusters describe <CLUSTER_NAME> \
  --region <REGION> \
  --format="value(privateClusterConfig.masterIpv4CidrBlock)"

Add a firewall rule:

gcloud compute firewall-rules create cert-manager-9443 \
  --source-ranges <GKE_CONTROL_PLANE_CIDR> \
  --target-tags <GKE_CONTROL_PLANE_TAG> \
  --allow TCP:9443

GKE Autopilot — Known Limitations

Auto language detection: NOT supported
UI-driven instrumentation toggle: NOT supported
Must use mw-autoinstrumentation.mode=manual with manual annotations
features.enableHostMonitoring must be set to false

EKS Fargate — Known Limitations

DaemonSets: NOT supported
UI-driven auto-instrumentation: NOT supported (use manual annotations)
mw-agent-ns must be included in a Fargate profile or pods stay in Pending
Logs collected via AWS Fluent Bit log router (not /var/log/containers)

11. OTel CRD Conflicts

If install fails with the error:

Error: INSTALLATION FAILED: unable to continue with install:
CustomResourceDefinition "targetallocators.opentelemetry.io" exists
and cannot be imported into the current release

Option 1 — Remove old CRDs (Recommended):

kubectl delete crd \
  opentelemetrycollectors.opentelemetry.io \
  instrumentations.opentelemetry.io \
  targetallocators.opentelemetry.io \
  opampbridges.opentelemetry.io

Option 2 — Keep existing CRDs, skip creation:

--set mw-autoinstrumentation.opentelemetry-operator.crds.create=false

12. TLS and cert-manager Troubleshooting

Without cert-manager — use self-signed certs:

--set mw-autoinstrumentation.webhook.certManager.enabled=false
--set mw-autoinstrumentation.opentelemetry-operator.admissionWebhooks.certManager.enabled=false

After upgrades or cert expiry, restart these components:

kubectl rollout restart deploy/mw-agent-opentelemetry-operator -n mw-agent-ns
kubectl rollout restart deploy/mw-auto-injector -n mw-agent-ns

13. Diagnostic Script

Save as mw-debug.sh and run with bash mw-debug.sh:

#!/bin/bash
NS="mw-agent-ns"

echo "========================================="
echo "  MW-Agent Diagnostic Report"
echo "  $(date)"
echo "========================================="

echo ""
echo "--- Kubernetes Context ---"
kubectl config current-context

echo ""
echo "--- All Resources in $NS ---"
kubectl get all -n $NS

echo ""
echo "--- Pod Status ---"
kubectl get pods -n $NS -o wide

echo ""
echo "--- DaemonSet Status ---"
kubectl get ds -n $NS

echo ""
echo "--- Deployments ---"
kubectl get deploy -n $NS

echo ""
echo "--- Services ---"
kubectl get svc -n $NS

echo ""
echo "--- Instrumentation CR ---"
kubectl get otelinst -n $NS 2>/dev/null || echo "No Instrumentation CR found"

echo ""
echo "--- Recent Events (last 20) ---"
kubectl get events -n $NS --sort-by='.lastTimestamp' | tail -20

echo ""
echo "--- Pods NOT Ready ---"
kubectl get pods -n $NS --field-selector=status.phase!=Running 2>/dev/null

echo ""
echo "--- Helm Release ---"
helm status mw-agent -n $NS 2>/dev/null || echo "Helm release not found"

echo ""
echo "--- Helm Values ---"
helm get values mw-agent -n $NS 2>/dev/null || echo "No values found"

echo ""
echo "========================================="
echo "  Done. Share this output with the team."
echo "========================================="

13 Patch Autoinstrumenation Image

kubectl patch instrumentation mw-autoinstrumentation \
  -n mw-agent-ns \
  --type='merge' \
  -p '{
    "spec": {
      "go": {
        "image": "ghcr.io/middleware-labs/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.24.0"
      }
    }
  }'

kubectl patch instrumentation mw-autoinstrumentation -n mw-agent-ns --type='json' -p='[
  {
    "op": "add",
    "path": "/spec/java/env/-",
    "value": {
      "name": "YOUR_ENV_NAME",
      "value": "your-env-value"
    }
  }
]'

14 Patch resources remove deployment limits

kubectl patch deployment mw-kube-agent -n mw-agent-ns --type json -p '[
  {"op": "remove", "path": "/spec/template/spec/containers/0/resources/limits"}
]'

15 To see sidecar golang instrumentation logs

kubectl logs shippingservice-846bddc898-8nb4z -c opentelemetry-auto-instrumentation -n default

16 Find executable path for golang process

K8s Troubleshooting

ps aux
ls -l /proc/<PID>/exe

17. Uninstall and Cleanup

# Remove Helm release
helm uninstall mw-agent -n mw-agent-ns

# Delete namespace
kubectl delete namespace mw-agent-ns

18 OBI ebpf setup

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-app
  namespace: default
  labels:
    app: flask-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask-app
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: flask-app
    spec:
      containers:
        - name: flask-app
          image: ghcr.io/middleware-labs/node-redis-example:test54
          ports:
            - name: http
              containerPort: 8000
              protocol: TCP
          env:
            - name: ENVIRONMENT
              value: production
            - name: TZ
              value: America/Chicago
        - name: obi
          image: >-
            ghcr.io/open-telemetry/opentelemetry-ebpf-instrumentation/ebpf-instrument:main
          env:
            - name: OTEL_SERVICE_NAME
              value: flask-app
            # your main app container port
            - name: OTEL_EBPF_OPEN_PORT
              value: '8080'
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: http://mw-service.mw-agent-ns:9320
            - name: OTEL_EBPF_KUBE_METADATA_ENABLE
              value: 'false'
            - name: OTEL_EBPF_BPF_CONTEXT_PROPAGATION
              value: 'all'
          securityContext:
            privileged: true
      shareProcessNamespace: true
      securityContext: {}

19 If otel operator is already there

--set mw-autoinstrumentation.opentelemetry-operator.enabled=false

20 Debug pod

K8s Network Debug Pod Spin up a temporary debug pod with curl, dig, nslookup, netcat and other network tools pre-installed:

kubectl run debug --rm -it --image=nicolaka/netshoot -n <NAMESPACE> -- /bin/bash

Once inside the debug pod:

curl -v http://mw-service.mw-agent-ns:9320
nslookup mw-service.mw-agent-ns
dig mw-service.mw-agent-ns
nc -zv mw-service.mw-agent-ns 9320
ping mw-service.mw-agent-ns

21 Autoinstrument resource yaml

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: mw-autoinstrumentation
  namespace: mw-agent-ns
spec:
  exporter:
    endpoint: http://mw-service.mw-agent-ns:9319
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  python:
    env:
      - name: MW_AGENT_SERVICE
        value: mw-service.mw-agent-ns.svc.cluster.local
      - name: GIT_PYTHON_REFRESH
        value: quiet
      - name: MW_K8S_PYTHON_INSTRUMENTATION
        value: 'true'
      - name: MW_TRACKER
        value: 'false'
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://mw-service.mw-agent-ns:9320
      - name: MW_API_KEY
        value: your-api-key
      - name: OTEL_PROPAGATORS
        value: tracecontext,baggage,b3
  nodejs:
    env:
      - name: OTEL_NODE_DISABLED_INSTRUMENTATIONS
        value: fs,dns,net
      - name: OTEL_PROPAGATORS
        value: b3,tracecontext,baggage
  go:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://mw-service.mw-agent-ns:9319
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: grpc
      - name: OTEL_PROPAGATORS
        value: b3,tracecontext,baggage
  dotnet:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://mw-service.mw-agent-ns:9320
      - name: OTEL_PROPAGATORS
        value: b3,tracecontext,baggage
  java:
    env:
      - name: MW_AGENT_SERVICE
        value: mw-service.mw-agent-ns.svc.cluster.local
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://mw-service.mw-agent-ns:9319
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: grpc
      - name: MW_API_KEY
        value: your-api-key
      - name: OTEL_PROPAGATORS
        value: tracecontext,baggage,b3

22 Use Mw agent LTS version

helm install mw-agent middleware-labs/mw-kube-agent-v3 \
  --version 1.6.7 \
  --set global.mw.apiKey=api-key \
  --set global.mw.target=https://msehd-fdr.middleware.io:443 \
  --set opsai.enabled=true \
  --set mw-autoinstrumentation.enabled=true \
  --set global.clusterMetadata.name=old \
  -n mw-agent-ns --create-namespace

23 Huge Number of Resources

kubectl get all -A | wc -l

24 Force apply

helm upgrade mw-agent middleware-labs/mw-kube-agent-v3 \
  --reuse-values \
  --set daemonset.resources.limits.memory=1024Mi \
  -n mw-agent-ns \
  --server-side=false

25 Operator Debug

kubectl patch deployment mw-agent-opentelemetry-operator \
  -n mw-agent-ns \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value": "1Gi"}]'

kubectl patch deployment mw-agent-opentelemetry-operator \
-n mw-agent-ns \
--type='json' \
-p='[
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/args/-",
    "value": "--zap-log-level=debug"
  }
]'

MW K8S Agent Debugging Cheat Sheet

MW-Agent Debugging & Troubleshooting Cheat Sheet

1. Quick Health Check

2. Verify Core Agent Components

3. Verify Auto-Instrumentation Components

4. Verify OpsAI

5. Log Inspection

5.1 Core Agent Logs

5.2 Auto-Instrumentation Logs

5.3 Search for Errors

6. Network and Connectivity

7. Helm Debugging

8. Auto-Instrumentation Debugging

8.1 Apps Not Detected

8.2 Instrumentation Not Working

8.3 Manual Annotation Quick Reference

8.4 Multi-Container Pods

8.5 Namespace Scoping

9. Common Issues and Fixes

No data in Middleware dashboard

mw-kube-agent OOMKilled or crash-looping

DaemonSet not scheduled on all nodes

Multiple agents running

Apps not detected

Instrumentation not injecting

Webhook or cert errors

OTel CRD conflict on install

Config changes not taking effect

10. Provider-Specific Troubleshooting

GKE Private Cluster — Webhook Port 9443 Blocked

GKE Autopilot — Known Limitations

EKS Fargate — Known Limitations

11. OTel CRD Conflicts

12. TLS and cert-manager Troubleshooting

13. Diagnostic Script

13 Patch Autoinstrumenation Image

14 Patch resources remove deployment limits

15 To see sidecar golang instrumentation logs

16 Find executable path for golang process

17. Uninstall and Cleanup

18 OBI ebpf setup

19 If otel operator is already there

20 Debug pod

21 Autoinstrument resource yaml

22 Use Mw agent LTS version

23 Huge Number of Resources

24 Force apply

25 Operator Debug

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally