New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add go profile instrumentation to kubectl #68679

Open
dlespiau opened this Issue Sep 14, 2018 · 1 comment

Comments

Projects
None yet
2 participants
@dlespiau

dlespiau commented Sep 14, 2018

We've had some slowness talking to an API server from a certain host (18s vs 3s!) and the verbose logs, eg. -v9, weren't enough to debug what taking time. Fortunately golang offers nice profiling features and one can output various profiles (in this case I was interested in the block profile!), enabling to see more insight in what is taking time.

For instance, one can clearly see the effect of having /home/$user on an NFS mount when writing the kubectl cache at the default location.

A tentative PR will follow to add the ability to write pprof profiles to kubectl.

@dlespiau

This comment has been minimized.

Show comment
Hide comment
@dlespiau

dlespiau Sep 14, 2018

/sig cli
/kind feature

dlespiau commented Sep 14, 2018

/sig cli
/kind feature

dlespiau added a commit to dlespiau/kubernetes that referenced this issue Sep 14, 2018

Add go profile instrumentation to kubectl
This commit adds two new global options to kubectl: --profile and
--profile-output, writing out go profiles to disk to debug interesting and
unexpected kubectl behaviour.

As an example, here is how to capture a block file, eg. for how long are we
blocked on I/O and where?

$ kubectl get nodes --profile=mutex -v6
$ go tool pprof -png ./profile.pprof > out.png
$ google-chrome out.png

Fixes: #68679

@dlespiau dlespiau referenced a pull request that will close this issue Sep 14, 2018

Open

Add go profile instrumentation to kubectl #68681

dlespiau added a commit to dlespiau/kubernetes that referenced this issue Sep 14, 2018

Add go profile instrumentation to kubectl
This commit adds two new global options to kubectl: --profile and
--profile-output, writing out go profiles to disk to debug interesting and
unexpected kubectl behaviour.

As an example, here is how to capture a block file, eg. for how long are we
blocked on I/O and where?

$ kubectl get nodes --profile=mutex -v6
$ go tool pprof -png ./profile.pprof > out.png
$ google-chrome out.png

Fixes: #68679

dlespiau added a commit to dlespiau/kubernetes that referenced this issue Sep 17, 2018

Add go profile instrumentation to kubectl
This commit adds two new global options to kubectl: --profile and
--profile-output, writing out go profiles to disk to debug interesting and
unexpected kubectl behaviour.

As an example, here is how to capture a block file, eg. for how long are we
blocked on I/O and where?

$ kubectl get nodes --profile=mutex -v6
$ go tool pprof -png ./profile.pprof > out.png
$ google-chrome out.png

Fixes: #68679

dlespiau added a commit to dlespiau/kubernetes that referenced this issue Sep 18, 2018

Add go profile instrumentation to kubectl
This commit adds two new global options to kubectl: --profile and
--profile-output, writing out go profiles to disk to debug interesting and
unexpected kubectl behaviour.

As an example, here is how to capture a block file, eg. for how long are we
blocked on I/O and where?

$ kubectl get nodes --profile=mutex -v6
$ go tool pprof -png ./profile.pprof > out.png
$ google-chrome out.png

Fixes: #68679
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment