New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add go profile instrumentation to kubectl #68681

Merged
merged 1 commit into from Sep 26, 2018

Conversation

Projects
None yet
7 participants
@dlespiau
Contributor

dlespiau commented Sep 14, 2018

This commit adds two new global options to kubectl: --profile and
--profile-output, writing out go profiles to disk to debug interesting and
unexpected kubectl behaviour.

As an example, here is how to capture a block file, eg. for how long are we
blocked on I/O and where?

$ kubectl get nodes --profile=block -v6
$ go tool pprof -png ./profile.pprof > out.png
$ google-chrome out.png

What this PR does / why we need it:

This instrumentation is useful to debug abnormal kubectl behaviour or do performance work.

Fixes: #68679

kubectl has gained new --profile and --profile-output options to output go profiles

As an example here's a block profile for the kubectl get nodes command above, rebuilding its cache. The api server isn't local.

screenshot from 2018-09-14 16-29-33

@bboreham

This comment has been minimized.

Show comment
Hide comment
@bboreham

bboreham Sep 14, 2018

Contributor

/ok-to-test

Contributor

bboreham commented Sep 14, 2018

/ok-to-test

@neolit123

This comment has been minimized.

Show comment
Hide comment
@neolit123

neolit123 Sep 15, 2018

Member

/kind feature
up to the maintainers.

Member

neolit123 commented Sep 15, 2018

/kind feature
up to the maintainers.

@dlespiau

This comment has been minimized.

Show comment
Hide comment
@dlespiau

dlespiau Sep 17, 2018

Contributor

/retest

The pull-kubernetes-e2e-kops-aws failure seems like a flake.
The pull-kubernetes-verify test is still using the previous version of the PR:

I0914 16:31:40.316] Args: --job=pull-kubernetes-verify --service-account=/etc/service-account/service-account.json --upload=gs://kubernetes-jenkins/logs --job=pull-kubernetes-verify --repo=k8s.io/kubernetes=master:94e59f1636a8f8b1566ef35180519d4de5586b79,68681:8653e09382d6becd8a72d0bc06a47e7020008a2b --service-account=/etc/service-account/service-account.json --upload=gs://kubernetes-jenkins/pr-logs --timeout=75 --scenario=kubernetes_verify -- '--branch=${PULL_BASE_REF}' --script=./hack/jenkins/verify-dockerized.sh

8653e09 is the previous commit. Hoping /retest will do the right thing.

Edit: The bot edited the previous comment removing the pull-kubernetes-verify failure!
Edit2: Ah, now I get it, the ci bot only keep one comment with the latest test results, oops!
Edit3: All good!

Contributor

dlespiau commented Sep 17, 2018

/retest

The pull-kubernetes-e2e-kops-aws failure seems like a flake.
The pull-kubernetes-verify test is still using the previous version of the PR:

I0914 16:31:40.316] Args: --job=pull-kubernetes-verify --service-account=/etc/service-account/service-account.json --upload=gs://kubernetes-jenkins/logs --job=pull-kubernetes-verify --repo=k8s.io/kubernetes=master:94e59f1636a8f8b1566ef35180519d4de5586b79,68681:8653e09382d6becd8a72d0bc06a47e7020008a2b --service-account=/etc/service-account/service-account.json --upload=gs://kubernetes-jenkins/pr-logs --timeout=75 --scenario=kubernetes_verify -- '--branch=${PULL_BASE_REF}' --script=./hack/jenkins/verify-dockerized.sh

8653e09 is the previous commit. Hoping /retest will do the right thing.

Edit: The bot edited the previous comment removing the pull-kubernetes-verify failure!
Edit2: Ah, now I get it, the ci bot only keep one comment with the latest test results, oops!
Edit3: All good!

@apelisse

Looks good to me, it should either fail or default if no profile is specified though

Show outdated Hide outdated pkg/kubectl/cmd/profiling.go
Show resolved Hide resolved pkg/kubectl/cmd/profiling.go
@dlespiau

This comment has been minimized.

Show comment
Hide comment
@dlespiau

dlespiau Sep 18, 2018

Contributor

If no profile is specified, profileName is "" and those functions do nothing, which is the intended behaviour. If profileName is non zero, the profile name is validated by initProfiling to make sure it's the name of a valid profile and fails if it isn't. I think we're covered then.

Contributor

dlespiau commented Sep 18, 2018

If no profile is specified, profileName is "" and those functions do nothing, which is the intended behaviour. If profileName is non zero, the profile name is validated by initProfiling to make sure it's the name of a valid profile and fails if it isn't. I think we're covered then.

@dims

This comment has been minimized.

Show comment
Hide comment
@dims

dims Sep 18, 2018

Member

/uncc

Member

dims commented Sep 18, 2018

/uncc

@k8s-ci-robot k8s-ci-robot removed the request for review from dims Sep 18, 2018

@apelisse

This comment has been minimized.

Show comment
Hide comment
@apelisse

apelisse Sep 18, 2018

Member

If no profile is specified, profileName is "" and those functions do nothing

Yeah, I don't think I agree. Running kubectl --profile whatever shouldn't skip profiling, that's too confusing.

Member

apelisse commented Sep 18, 2018

If no profile is specified, profileName is "" and those functions do nothing

Yeah, I don't think I agree. Running kubectl --profile whatever shouldn't skip profiling, that's too confusing.

@dlespiau

This comment has been minimized.

Show comment
Hide comment
@dlespiau

dlespiau Sep 18, 2018

Contributor

Ah I see what you mean, I think that's a general problem with options though :)

$ kubectl --kubeconfig get nodes
Error: unknown command "nodes" for "kubectl"
Run 'kubectl --help' for usage.
unknown command "nodes" for "kubectl"

--profile alone has the same behaviour:

$ ./_output/bin/kubectl --profile get nodes
Error: unknown command "nodes" for "kubectl"
Run 'kubectl --help' for usage.
unknown command "nodes" for "kubectl"

--profile alone isn't meaningful, just like --kubeconfig alone isn't. The parser takes the next token as the --profile option and tries to find the nodes sub-command, which doesn't exist. I tried to play with -- to terminate the option list and force an empty --profile option but to no avail. --profile on its own is simply not valid, just like --kubeconfig on its own.

To achieve meaningful --profile option, we'd need to have it a boolean option I believe and the UX would become:

$ kubectl --profile --profile-kind=block --profile-output=foo.pprof get nodes

with --profile-kind defaulting to, say, cpu. I have a preference for only having two options.

Just in case the whatever in your example meant a bogus profile name, the current code errors out:

$ ./_output/bin/kubectl --profile whatever get nodes
[...]
unknown profile 'whatever'
Contributor

dlespiau commented Sep 18, 2018

Ah I see what you mean, I think that's a general problem with options though :)

$ kubectl --kubeconfig get nodes
Error: unknown command "nodes" for "kubectl"
Run 'kubectl --help' for usage.
unknown command "nodes" for "kubectl"

--profile alone has the same behaviour:

$ ./_output/bin/kubectl --profile get nodes
Error: unknown command "nodes" for "kubectl"
Run 'kubectl --help' for usage.
unknown command "nodes" for "kubectl"

--profile alone isn't meaningful, just like --kubeconfig alone isn't. The parser takes the next token as the --profile option and tries to find the nodes sub-command, which doesn't exist. I tried to play with -- to terminate the option list and force an empty --profile option but to no avail. --profile on its own is simply not valid, just like --kubeconfig on its own.

To achieve meaningful --profile option, we'd need to have it a boolean option I believe and the UX would become:

$ kubectl --profile --profile-kind=block --profile-output=foo.pprof get nodes

with --profile-kind defaulting to, say, cpu. I have a preference for only having two options.

Just in case the whatever in your example meant a bogus profile name, the current code errors out:

$ ./_output/bin/kubectl --profile whatever get nodes
[...]
unknown profile 'whatever'
@apelisse

This comment has been minimized.

Show comment
Hide comment
@apelisse

apelisse Sep 18, 2018

Member

Sorry, I made a mistake ... I meant $ kubectl --profile= get pods. It's still weird if this does no profiling.

Member

apelisse commented Sep 18, 2018

Sorry, I made a mistake ... I meant $ kubectl --profile= get pods. It's still weird if this does no profiling.

@apelisse

This comment has been minimized.

Show comment
Hide comment
@apelisse

apelisse Sep 18, 2018

Member

Sorry, I made a mistake ... I meant $ kubectl --profile= get pods. It's still weird if this does no profiling.

or maybe it's not, maybe you should have an option named "none", and make it the default?

Member

apelisse commented Sep 18, 2018

Sorry, I made a mistake ... I meant $ kubectl --profile= get pods. It's still weird if this does no profiling.

or maybe it's not, maybe you should have an option named "none", and make it the default?

Add go profile instrumentation to kubectl
This commit adds two new global options to kubectl: --profile and
--profile-output, writing out go profiles to disk to debug interesting and
unexpected kubectl behaviour.

As an example, here is how to capture a block file, eg. for how long are we
blocked on I/O and where?

$ kubectl get nodes --profile=mutex -v6
$ go tool pprof -png ./profile.pprof > out.png
$ google-chrome out.png

Fixes: #68679
@dlespiau

This comment has been minimized.

Show comment
Hide comment
@dlespiau

dlespiau Sep 18, 2018

Contributor

Oh I like the "none" idea! Thanks for taking the time to think about it and make a great suggestion!

Now, giving a "" string errors out as such:

./_output/bin/kubectl --profile= get nodes
[...]
unknown profile ''

Updated the code and help string with `"none"``.

Contributor

dlespiau commented Sep 18, 2018

Oh I like the "none" idea! Thanks for taking the time to think about it and make a great suggestion!

Now, giving a "" string errors out as such:

./_output/bin/kubectl --profile= get nodes
[...]
unknown profile ''

Updated the code and help string with `"none"``.

@apelisse

This comment has been minimized.

Show comment
Hide comment
@apelisse

apelisse Sep 18, 2018

Member

/lgtm

Thanks!

Member

apelisse commented Sep 18, 2018

/lgtm

Thanks!

@k8s-ci-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-ci-robot

k8s-ci-robot Sep 18, 2018

Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: apelisse, dlespiau

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Contributor

k8s-ci-robot commented Sep 18, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: apelisse, dlespiau

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@dlespiau

This comment has been minimized.

Show comment
Hide comment
@dlespiau

dlespiau Sep 19, 2018

Contributor

Thanks a lot for the review!

Contributor

dlespiau commented Sep 19, 2018

Thanks a lot for the review!

@seans3

This comment has been minimized.

Show comment
Hide comment
@seans3

seans3 Sep 20, 2018

Collaborator

Would it be possible to add a unit test (e.g. profile_test.go)?

Collaborator

seans3 commented Sep 20, 2018

Would it be possible to add a unit test (e.g. profile_test.go)?

@dlespiau

This comment has been minimized.

Show comment
Hide comment
@dlespiau

dlespiau Sep 21, 2018

Contributor

Thinking a bit about it, we could maybe add tests along the lines of:

  • Check that giving a bogus profile name leads initProfiling to return an error
  • Check that giving a valid profile name generates a file (and maybe even check that is indeed a pprof file). Maybe cycle through all profile names to check all well-known names generate profiles.
Contributor

dlespiau commented Sep 21, 2018

Thinking a bit about it, we could maybe add tests along the lines of:

  • Check that giving a bogus profile name leads initProfiling to return an error
  • Check that giving a valid profile name generates a file (and maybe even check that is indeed a pprof file). Maybe cycle through all profile names to check all well-known names generate profiles.

@k8s-ci-robot k8s-ci-robot merged commit cc04abc into kubernetes:master Sep 26, 2018

18 checks passed

cla/linuxfoundation dlespiau authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Skipped
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-e2e-kubeadm-gce Skipped
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment