Debug Pilot with Squash Delve

Welcome to the istio wiki!

Prerequisites:

• A kubernetes cluster is running properly.
• Golang and delve has been properly installed.
• istio is installed and running properly in the kubernetes cluster
• istio source is downloaded, and following the instructions, make sure istio and its containers can be properly built.
• if you have multiple worker nodes in the cluster, you'd need a docker registry to push your built containers so that they can be used in your cluster. The docker hub is free to use, and fairly easy to setup.
• squash is installed and verified working.

Click on the above links and follow the instructions to install the required software. In this guide, all the required software are installed on the cluster's master node. Although all the containers can be debugged with the same procedure, we'll use pilot as an example in this guide.

Build Pilot With Debugger Info and Right Base Image

The default istio build will not include debugger info in the image. The the time being, you can apply the following patch in your istio repo before starting your build:

diff --git a/pilot/docker/Dockerfile.pilot b/pilot/docker/Dockerfile.pilot
index 473152f..a9c6bb6 100644
--- a/pilot/docker/Dockerfile.pilot
+++ b/pilot/docker/Dockerfile.pilot
@@ -1,3 +1,3 @@
-FROM scratch
+FROM alpine
ADD pilot-discovery /usr/local/bin/
ENTRYPOINT ["/usr/local/bin/pilot-discovery"]

Squash requires the resulting docker image to contain the mnt directory. Therefore, the base image must satisfy that requirement.

I'm using the following little script build.sh to build and push the resulting pilot image to docker.io/baodongli. You can adapt it to your needs.

• cd
• build.sh

export TAG=$1
make DEBUG=1 pilot
sudo make DEBUG=1 docker.pilot
sudo docker tag pilot:latest baodongli/pilot:$TAG
sudo docker push baodongli/pilot:$TAG

Debug Pilot

• List the squash pods

$ kubectl get pods -n squash
NAME                           READY     STATUS    RESTARTS   AGE
squash-client-mb7cv            1/1       Running   1          3d
squash-client-pdfks            1/1       Running   1          3d
squash-client-wjj8w            1/1       Running   0          3d
squash-server-8dd6b6bf-sv9s5   1/1       Running   1          2d

‍‍ ‍‍‍‍‍* Edit istio pilot deployment

$ kubectl edit deployment -n istio-system istio-pilot

• With the deployment file opened by the editor, search for the container discovery that's running the pilot-discovery process. Replace its image file with the one just built. Also update imagePullPolicy to be Always.
• Save and exit the editor.
• List the istio pods.

$ kubectl get pods -n istio-system
NAME                             READY     STATUS    RESTARTS   AGE
istio-ca-78b58d7b5d-rmz6k        1/1       Running   0          4h
istio-ingress-78545c5bc9-2b8t5   1/1       Running   0          4h
istio-mixer-6f7db9c9fc-tnvdt     3/3       Running   0          4h
istio-pilot-85bdbfc4dd-8qv9c     2/2       Running   0          57s

• invoke the squash command debug-container which has the format as below:

squash --url=http://localhost:8001/api/v1/namespaces/squash/services/squash-server:http-squash-api/proxy/api/v2 debug-container --namespace <namespace> <docker-image-name> <pod-name> <container-name> dlv [--processName <process-name>]

The above command sends a request to squash to debug a container that is running inside the pod in the namespace . The container's image name must be specified that is matching exactly with the one used by the running container. Squash finds which worker node the container is running on and the process id. The squash client running on that worker node will then attach to the process. Optionally, the process name to be debugged can be specified if there are multiple processes running in the container.

• In the case istio pilot is running in the namespace istio-system as shown above, the command line is shown in below. The command returns a debug config id if successful.

$ squash --url=http://localhost:8001/api/v1/namespaces/squash/services/squash-server:http-squash-api/proxy/api/v2 debug-container --namespace istio-system docker.io/baodongli/pilot:0.4.0 istio-pilot-85bdbfc4dd-8qv9c discovery dlv
Debug config id: x9os9l5zFE

• list squash attachments If the debug session is successfully launched, the list command will display where the debug session is running by providing the debugger address. Any state other than attached (or probably waiting) may indicate failure in attaching.

$ squash --url=http://localhost:8001/api/v1/namespaces/squash/services/squash-server:http-squash-api/proxy/api/v2 list 
attachments
State    |ID         |Debugger |Image                           |Debugger Address
attached |x9os9l5zFE |dlv      |docker.io/baodongli/pilot:0.4.0 |squash-client-wjj8w.squash:38611‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
‍```

* Notice that with the debug id x9os9l5zFE, the debugger address is squash-client-wjj8w.squash:38611. squash-client-wjj8w is one of the squash client pods listed in above. Create a port-forward for it as in below. The port-forward allows connecting to the debug session.

$ kubectl port-forward -n squash squash-client-wjj8w 38611& [3] 7378 Forwarding from 127.0.0.1:38611 -> 38611

• connect to the debug instance, and break at discovery.go line 477. then hit continue

$ dlv connect 127.0.0.1:38611
Handling connection for 38611
Type 'help' for list of commands.
(dlv) break discovery.go:477
Breakpoint 1 set at 0x1790d82 for istio.io/istio/pilot/proxy/envoy.(*DiscoveryService).ListAllEndpoints() ./pilot/proxy/envoy/discovery.go:477
(dlv) continue

• on a different xterm, enter the following commands. The commands sends a sds query to pilot.

$ cd istio.io/istio/bin
$ istio-proxy-cfg sds

• back to the debug xterm, dlv will stop the pilot at the designated line as in below:

(dlv) c
> istio.io/istio/pilot/proxy/envoy.(*DiscoveryService).ListAllEndpoints() ./pilot/proxy/envoy/discovery.go:477 (hits goroutine(845):1 total:1) (PC: 0x1790d82)
 472: ds.ldsCache.clear()
 473: }
 474:
 475: // ListAllEndpoints responds with all Services and is not restricted to a single service-key
 476: func (ds *DiscoveryService) ListAllEndpoints(_ *restful.Request, response *restful.Response) {
=> 477: methodName := "ListAllEndpoints"
 478: incCalls(methodName)
 479:
 480: services := make([]*keyAndService, 0)
 481:
 482: svcs, err := ds.Services()
(dlv) 

• Afterwards, you can issue delve commands to step through, add additional breakpoints, examine stack, inspect variables, etc. At the end, enter the continue command.

Notes

• An issue is documented in here: Can't attach to the container after exiting the current debug session.
• Due to the event-driven nature of pilot, you may not need to manually trigger the breakpoints to be hit. For example, if you set a breakpoint for the ListClusters method in discovery.go, it will be hit constantly.
• Since Pilot maintains a response cache, you may hit a breakpoint only once with a xDS request. It won't break any more with the following same requests until some change causes the response cache to be evicted.
• A few advantages of using squash: ** You don't need to install go and delve on every worker nodes yourself ** debug filter in envoy can be configured to trigger the debug-container attachment by sending a request to the microservice with a squash header. Note that this feature hasn't been fully supported yet.
• Alternative to using squash, you can install go and delve on every of your worker nodes yourself. You can log on to the node where your container to be debugged is running, and attach to it yourself.