Troubleshooting Istio

Troubleshooting Istio

Sidecar Injection

If your pods are failing to start, look into the MutatingAdmissionWebhook istio-sidecar-injector. When a pod is created, the Kubernetes api-server will call the sidecar injector service (Istiod). Errors during injection, or failure to connect to the service, can result in pods not being created.

Collecting Information

The replica set will generally contain any error messages. Gather this information with kubectl describe replicaset REPLICA_SET > replicaset.txt.

To get logs from Istiod, run: kubectl logs -n istio-system -l app=istiod --tail=100000000 > istiod.log.

To get the injection template: kubectl -n istio-system get configmap istio-sidecar-injector -o jsonpath={.data.config} > template.yaml

Istiod

Collecting information

To capture logs: kubectl logs -n istio-system -l app=istiod --tail=100000000 -c discovery > istiod.log.

To capture mesh config: kubectl get configmap -n istio-system -o jsonpath={.data.mesh} istio > meshconfig.yaml

To capture a proxy config dump from Istiod perspective: kubectl exec ISTIOD_POD -- curl 'localhost:8080/debug/config_dump?proxyID=POD_NAME.POD_NAMESPACE',

Capture a snapshot of the Istio Control Plane dashboard. Prefer this to a screenshot if possible, as it allows zooming, etc.

Performance Issues

If you are experiencing performance issues with Istiod, such as excessive CPU or memory usage, memory leaks, etc, it is helpful to capture profiles. Please see this page for help.

Sidecar/Gateway problems

Collecting Information

To get configuration and stats from a proxy (gateway or sidecar):

• Stats: kubectl exec $POD -c istio-proxy -- curl 'localhost:15000/stats' > stats
• Config Dump: kubectl exec $POD -c istio-proxy -- curl 'localhost:15000/config_dump' > config_dump.json
• Clusters Dump: kubectl exec $POD -c istio-proxy -- curl 'localhost:15000/clusters' > clusters
• Logs: kubectl logs $POD -c istio-proxy > proxy.log

To enable debug logging, which may be useful if the default log does not provide enough information:

• At runtime: istioctl proxy-config log POD --level=debug
• For a pod, set annotation: sidecar.istio.io/logLevel: "debug"
• For the whole mesh, install with --set values.global.proxy.logLevel=debug

To enable access logging, which may be useful to debug traffic, see here. More info about access log format can be found in Envoy docs.

Performance Issues

See Analyzing Istio Performance

Common Issues

• gRPC config stream closed: 13 in proxy logs, every 30 minutes. This error message is expected, as the connection to Pilot is intentionally closed every 30 minutes.
• gRPC config stream closed: 14 in proxy logs. If this occurs repeatedly it may indicate problems connecting to Pilot. However, a single occurance of this is typical when Envoy is starting or restarting.