Ability to enable profiling in Antrea components #1434
Comments
antoninbas
added
kind/feature
|
@tnqn I think this is something that would be very helpful to have in 0.11. I can take care of it if no one else has cycles, but let me see if you see any issue with the approach first. I have tested it or written any code yet. |
|
@weiqiangt it does look like profiling is enabled by default. Could you look into it if you have the chance and maybe share some steps? It would be great to document it. There is probably some authentication required? |
|
Sure. # antrea-controller/antrea-agent
TARGET=""
TOKEN=$(kubectl exec -it $(kubectl get pods -nkube-system -owide|grep antrea-controller|awk '{print $2}') -nkube-system -- cat /var/run/antrea/apiserver/loopback-client-token)
# ip address of the agent/controller
IP=""
# port number of the apiserver of the agent/controller
PORT=""
# heap/profile(for CPU sampling)/goroutines
CATEGORY=""
OUTPUT="filename"
wget --header="Authorization: Bearer $TOKEN" https://$IP:$PORT/debug/pprof/$CATEGORY --no-check-certificate -O $OUTPUTFor documenting, where would you like to put them in? |
|
Thanks for the information @weiqiangt. This document would be the right place: https://github.com/vmware-tanzu/antrea/blob/master/docs/troubleshooting.md However it's not as simple as I had hoped, and it seems that it would not be possible to use |
|
Yes, your idea is nice. |
|
I can take a look at it later this week. I think we may be able to use the |
With this command antctl can operate as a reverse proxy for Antrea APIs, similarly to "kubectl proxy" for the K8s APIs (we rely on the k8s.io/kubectl Go module for the implementation). Thanks to this, troubleshooting the APIs can become much easier and we hide complexity from clients. One example is that it becomes much easier to use "go tool pprof". The drawback here is that the HTTPS connection between the proxy and the Antrea Agent / Controller is not secure (this can be fixed for the Controller at least); in this it is very similar to the supportbundle command implementation. Fixes antrea-io#1434 TODO: documentation, tests
With this command antctl can operate as a reverse proxy for Antrea APIs, similarly to "kubectl proxy" for the K8s APIs (we rely on the k8s.io/kubectl Go module for the implementation). Thanks to this, troubleshooting the APIs can become much easier and we hide complexity from clients. One example is that it becomes much easier to use "go tool pprof". The drawback here is that the HTTPS connection between the proxy and the Antrea Agent / Controller is not secure (this can be fixed for the Controller at least); in this it is very similar to the supportbundle command implementation. Fixes antrea-io#1434 TODO: documentation
With this command antctl can operate as a reverse proxy for Antrea APIs, similarly to "kubectl proxy" for the K8s APIs (we rely on the k8s.io/kubectl Go module for the implementation). Thanks to this, troubleshooting the APIs can become much easier and we hide complexity from clients. One example is that it becomes much easier to use "go tool pprof". The drawback here is that the HTTPS connection between the proxy and the Antrea Agent / Controller is not secure (this can be fixed for the Controller at least); in this it is very similar to the supportbundle command implementation. Fixes antrea-io#1434
With this command antctl can operate as a reverse proxy for Antrea APIs, similarly to "kubectl proxy" for the K8s APIs (we rely on the k8s.io/kubectl Go module for the implementation). Thanks to this, troubleshooting the APIs can become much easier and we hide complexity from clients. One example is that it becomes much easier to use "go tool pprof". The drawback here is that the HTTPS connection between the proxy and the Antrea Agent / Controller is not secure (this can be fixed for the Controller at least); in this it is very similar to the supportbundle command implementation. Fixes antrea-io#1434
With this command antctl can operate as a reverse proxy for Antrea APIs, similarly to "kubectl proxy" for the K8s APIs (we rely on the k8s.io/kubectl Go module for the implementation). Thanks to this, troubleshooting the APIs can become much easier and we hide complexity from clients. One example is that it becomes much easier to use "go tool pprof". The drawback here is that the HTTPS connection between the proxy and the Antrea Agent / Controller is not secure (this can be fixed for the Controller at least); in this it is very similar to the supportbundle command implementation. Fixes #1434
Describe the problem/challenge you have
When a user runs into an issue like this one, it is hard to collect relevant information to troubleshoot the issue.
Describe the solution you'd like
We should have a way to enable profiling and the pprof HTTP server (https://golang.org/pkg/net/http/pprof/) for the Antrea components (Agent & Controller) after Antrea has been deployed in a cluster. We could consider having a config parameter for this. However updating the ConfigMap would require a restart of the Agent / Controller. My favorite solution would be to be able to use
antctlto enable profiling. This means that profiling could be enabled selectively for a specific Agent or the Controller (by exec'ing into the Pod and running antctl). It could look like this:Given that the antrea-agent Pod and the antrea-controller Pod both use the host network, I believe no changes to the Pod specs would be needed to enable access to the HTTP server using one of the Node's IP.
Anything else you would like to add?
Maybe there is some integration possible with
antctl support-bundle, but I don't think it's necessary at first.The text was updated successfully, but these errors were encountered: