Troubleshooting Istio
If your pods are failing to start, look into the MutatingAdmissionWebhook
istio-sidecar-injector
. When a pod is created, the Kubernetes api-server will call the sidecar injector service. Errors during injection, or failure to connect to the service, can result in pods not being created.
The replica set will generally contain any error messages. Gather this information with kubectl describe replicaset REPLICA_SET > replicaset.txt
.
To get logs from the sidecar injector, run: kubectl logs -n istio-system -l istio=sidecar-injector --tail=100000000 > injector.log
.
To get the injection template: kubectl -n istio-system get configmap istio-sidecar-injector -o jsonpath={.data.config} > template.yaml
To capture logs: kubectl logs -n istio-system -l istio=pilot --tail=100000000 -c discovery > pilot.log
.
To capture mesh config: kubectl get configmap -n istio-system -o jsonpath={.data.mesh} istio > meshconfig.yaml
Capture a snapshot of the Pilot dashboard. Prefer this to a screenshot if possible, as it allows zooming, etc.
If you are experiencing performance issues with Pilot, such as excessive CPU or memory usage, memory leaks, etc, it is helpful to capture profiles. Please see this page for help.
-
Configuration not synced: first push for [...] not received
in Pilot logs. This indicates Pilot did not recieve configuration from Galley. This can occur if Galley is not healthy or if the CRD is missing from the cluster.
To get configuration and stats from a proxy (gateway or sidecar):
kubectl exec $POD -c istio-proxy -- curl 'localhost:15000/stats' > stats
kubectl exec $POD -c istio-proxy -- curl 'localhost:15000/config_dump' > config_dump.json
kubectl exec $POD -c istio-proxy -- curl 'localhost:15000/clusters' > clusters
kubectl logs $POD -c istio-proxy > proxy.log
To enable debug logging, which may be useful if the default log does not provide enough information:
kubectl exec $POD -c istio-proxy -- curl -X POST 'localhost:15000/logging?level=debug'
See Analyzing Istio Performance
-
gRPC config stream closed: 13
in proxy logs, every 30 minutes. This error message is expected, as the connection to Pilot is intentionally closed every 30 minutes. -
gRPC config stream closed: 14
in proxy logs. If this occurs repeatedly it may indicate problems connecting to Pilot. However, a single occurance of this is typical when Envoy is starting or restarting.
Visit istio.io to learn how to use Istio.
- Preparing for Development Mac
- Preparing for Development Linux
- Troubleshooting Development Environment
- Repository Map
- GitHub Workflow
- Github Gmail Filters
- Using the Code Base
- Developing with Minikube
- Remote Debugging
- Verify your Docker Environment
- Istio Test Framework
- Working with Prow
- Test Grid
- Code Coverage FAQ
- Writing Good Integration Tests
- Test Flakes
- Release Manager Expectations
- Preparing Istio Releases
- 1.5 Release Information
- 1.6 Release Information
- 1.7 Release Information
- 1.8 Release Information
- 1.9 Release Information
- 1.10 Release Information
- 1.11 Release Information
- 1.12 Release Information
- 1.13 Release Information
- 1.14 Release Information
- 1.15 Release Information
- 1.16 Release Information
- 1.17 Release Information
- 1.18 Release Information
- 1.19 Release Information
- 1.20 Release Information
- 1.21 Release Information
- 1.22 Release Information
- Collecting Logs and Debug Info
- Dependency FAQ
- Working with discuss.istio.io
- Developing with and hosting upon OpenShift
- Adapter Dev Guide
- Adapter Walkthrough
- Attribute Generating Adapter Walkthrough
- Route Directive Adapter Development Guide
- Out of Tree Adapter Walkthrough
- Running a Local Instance
- Template Dev Guide
- Using a Custom Adapter
- Publishing Adapters and Templates to istio.io
- Enabling Envoy Authorization Service and gRPC Access Log Service With Mixer