-
Notifications
You must be signed in to change notification settings - Fork 12
tada apigee hybrid tcpdump
- TADA -- Troubleshooting And Debugging for Apigee
#apigee-hybrid #1.3.x #tcpdump #kubernetes-networking
For troubleshooting connection between Target endpoint and the backend itself, most problems with Apigee Proxies are solved by using the TRACE tool of the Edge UI.
More complex problems require usage of a Debug logging
The most complex problems call for usage of tcpdump. For Apigee hybrid, the scenario is complicated by the fact that apigee-runtime container runs as an apigee:apigee user. Good for your security, not so for tcpdump, which require root's permissions.
The following walkthrough uses Brian O'Connell's tip to find a Kubernetes container's interface at a cluster node How to get tcpdump for containers inside Kubernetes pods.
Problem: A backend server required a Host header setup, because DNS is not configured, we have an IP address-based URL.
We can use either JavaScript policy or AssignMessage policy to set up a Host header value. But we cannot be sure for 100% it worked, because as it happens now, Get Curl
button does not show the transformed request and response body is not shown under certain conditions, which might leave some space for speculation.
One of those curtain conditions is when Apigee response parser is unable to parse a response body. In this case, we are not going to see it.
Let's use http:/httpbin.org for our troubleshooting session.
?. The httpbin.org/get request conveniently returns sent headers
curl http://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "httpbin.org",
"User-Agent": "curl/7.64.1",
"X-Amzn-Trace-Id": "Root=1-5fec44cc-56672a2f419e9c67426910af"
},
"origin": "86.140.110.147",
"url": "http://httpbin.org/get"
}
?. Find out IP address for httpbin.org FQDN
ping httpbin.org
PING httpbin.org (3.211.1.78): 56 data bytes
...
?. Request to httpbin.org using IP address, returns expected host header containing IP address
curl http://3.211.1.78/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "3.211.1.78",
"User-Agent": "curl/7.64.1",
"X-Amzn-Trace-Id": "Root=1-5fec4571-02fe421e52942a49105014c3"
},
"origin": "86.140.110.147",
"url": "http://3.211.1.78/get"
}
?. Using +PROXY
/Reverse proxy
master, create an httpbin proxy:
Name: httpbin
Target: http://3.211.1.78
Next
Pass through (no authorization)
Next
Select checkbox: Deployment: test
Create
In a general case, you will have multiple apigee-runtime pods in multiple regions. As we want to intercept traffic to a specific single container, we need to send the request to the specific pod that hosts this container.
See for details: https://github.com/apigee/ahr/wiki/Hybrid-Ingress-Walkthrough-1.3
?. Verify that you're authenticated into a correct cluster
kubectl config get-contexts
?. Pick a container to work with. We are selecting first one from the list of eligible pods
export POD=$(kubectl get pods -n apigee |grep apigee-runtime |awk 'NR==1{print $1}'); echo $POD
apigee-runtime-emea-cs-hybrid-demo6-test-v110-2lsbg
?. Exec into a $POD
kubectl exec -it $POD --namespace apigee -c apigee-runtime -- sh
WARNING: There is no curl installed into an apigee-runtime pod prior to hybrid 1.3.x. Use
kubectl get pods -n apigee -o wide
to get pod IP address
kubectl run -it busyboxplus --image=radial/busyboxplus --restart=Never
to log into busyboxplus and run curl.
kubectl delete pod busyboxplus
to delete a stopped pod.
?. Get Pod IP value
kubectl get pod $POD -n apigee -o jsonpath='{.status.podIP}'
10.44.5.57
?. In the pod at the prompt, execute a request
export RUNTIME_IP=10.44.5.57
export RUNTIME_HOST_ALIAS=emea-cs-hybrid-demo6-test-dc1.hybrid-apigee.net
curl -k https://$RUNTIME_HOST_ALIAS:8443/httpbin/get --resolve "$RUNTIME_HOST_ALIAS:8443:$RUNTIME_IP"
Output:
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "3.211.1.78",
"User-Agent": "curl/7.35.0",
"X-Amzn-Trace-Id": "Root=1-5fec5cd8-4023c6b4521223404a5e5727"
},
"origin": "35.196.107.149",
"url": "http://3.211.1.78/get"
}
For details: How to get tcpdump for containers inside Kubernetes pods https://community.pivotal.io/s/article/How-to-get-tcpdump-for-containers-inside-Kubernetes-pods?language=en_US
?. Find the container name and node where your pod is deployed
kubectl describe pod $POD -n apigee|grep -E "Node:|Containers:|Container ID:|apigee-runtime:"
Node: gke-dc1-cluster-apigee-runtime-f80b30bd-btxy/10.142.0.19
Init Containers:
Container ID: docker://f551fe5cf7c23b138966d656fe7891bd620a62c7181d8b0e7f56817f578f31fd
Containers:
apigee-runtime:
Container ID: docker://95f76d7ffb881d19ff77117f3f9a657a91843c25ab49009cf5e9b71a3e1af97b
Image: google/apigee-runtime:1.1.0
?. Using Node: field value, find the node's region
gcloud compute instances list
...
gke-dc1-cluster-apigee-runtime-f80b30bd-btxy us-east1-b n1-standard-4 10.142.0.19 35.196.107.149 RUNNING
...
?. Ssh into the node
gcloud compute ssh gke-dc1-cluster-apigee-runtime-f80b30bd-btxy --zone us-east1-b
yuriyl@gke-dc1-cluster-apigee-runtime-f80b30bd-btxy ~ $
?. Using container Id, find the pod's unique network interface index inside it's container.
export CID=95f76d7ffb881d19ff77117f3f9a657a91843c25ab49009cf5e9b71a3e1af97b
docker exec $CID /bin/bash -c 'cat /sys/class/net/eth0/iflink'
Output:
60
?. Take the result from that and locate that interface on the worker
for i in /sys/class/net/veth*/ifindex; do grep -l 60 $i; done
Output:
/sys/class/net/veth052c835c/ifindex
?. Install toolbox at the node
As GKE nodes use COS, install and run the toolbox. For details: https://cloud.google.com/container-optimized-os/docs/how-to/toolbox
/usr/bin/toolbox
?. After logging into the toolbox, install tcpdump
apt-get update && apt-get install -y tcpdump
We need two terminal sessions:
- one to run tcpdump
- one to execute curl requests.
Using interface id from the previous section, capture incoming and outgoing traffic for httpbin.org IP address we found earlier
?. Start tcpdump to capture output into a request.pcap file.
tcpdump -i veth052c835c host 3.211.1.78 -w request.pcap
?. Execute request using curl in a separate session
?. Press Ctrl+C to interrupt the tcp dump capture.
Output:
^C9 packets captured
9 packets received by filter
0 packets dropped by kernel
?. Copy capture file out of toolbox to the node See also: https://cloud.google.com/container-optimized-os/docs/how-to/toolbox#getting_files_into_and_out_of_toolbox
?. Exit from toolbox session. Notice the container name.
root@gke-dc1-cluster-apigee-runtime-f80b30bd-btxy:~# exit
logout
Container yuriyl-gcr.io_google-containers_toolbox-20180918-00 exited successfully.
?. At the node, execute sudo cp
to copy request.pcap file from container to the node file system
sudo cp /var/lib/toolbox/yuriyl-gcr.io_google-containers_toolbox-20180918-00/root/request.pcap .
?. Exit the node and copy request.pcap file to your work machine
exit
?. Copy capture file to your work machine with installed wireshark.
gcloud compute scp gke-dc1-cluster-apigee-runtime-f80b30bd-btxy:~/request.pcap . --zone us-east1-b
?. Open the file in wireshark, select the GET line and right-click follow the HTTP stream.
?. Using Edge UI, add the following Assign Message Policy to the Target Endpoint PreFlow event.
<AssignMessage async="false" continueOnError="false" enabled="true" name="AssignMessage-SetHost">
<AssignVariable>
<Name>target.header.host</Name>
<Value>xxx.yyy.com</Value>
</AssignVariable>
<AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>
?. Save and Deploy a new version of the httpbin API proxy
?. Repeat tcpdump capture and analysis for a new curl request
Observe that the request now uses an overridden SNI-compliant host value.
- TADA: Apigee Hybrid Container Traffic Analysis with tcpdump for Target Requests
- TADAA: Cloud Code IDE Java Callout Debugging
- Hybrid Ingress Walkthrough 1.5
- Hybrid CRD Objects Diagram 1.3.2
- List of Components for Air-gapped Deployments
- GKE/EKS Multi-cloud Small Topology 1.4 [single-project] | AHR Profile Quick Start
- Single-Zone Cluster, Small footprint 1.1
- Multi-Zone Cluster, Large footprint 1.1
- Multi-Region Cluster, Large footprint 1.1
-
Private Cluster
-
Performance Testing: distGatling
- Delete Hybrid Installation
- Hybrid Ingress Walkthrough 1.3