New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

getPids - don't recursively traverse every dir in /proc #66367

Merged
merged 1 commit into from Oct 29, 2018

Conversation

@cpuguy83
Contributor

cpuguy83 commented Jul 19, 2018

What this PR does / why we need it:

filepath.Walk recursively traverses every dir, which is not what is
needed for getPids.
Instead only read the list of dirs in the top level of /proc.

benchmark              old ns/op     new ns/op     delta
BenchmarkGetPids-4     868684        195522        -77.49%
Speedup process lookup in /proc

@cpuguy83 cpuguy83 changed the title from PidOf - don't recursively traverse every dir to getPids - don't recursively traverse every dir Jul 19, 2018

@cpuguy83 cpuguy83 changed the title from getPids - don't recursively traverse every dir to getPids - don't recursively traverse every dir in /proc Jul 19, 2018

@k8s-ci-robot

This comment has been minimized.

Contributor

k8s-ci-robot commented Jul 19, 2018

@lanchongyizu: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/ok-to-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@xiaoxubeii

This comment has been minimized.

Member

xiaoxubeii commented Jul 19, 2018

/ok-to-test

}
// The bytes we read have '\0' as a separator for the command line
parts := bytes.SplitN(cmdline, []byte{0}, 2)
if len(parts) == 0 {

This comment has been minimized.

This comment has been minimized.

@cpuguy83

cpuguy83 Jul 19, 2018

Contributor

Good catch.

@@ -140,12 +142,10 @@ func getPids(re *regexp.Regexp) []int {
}
getPids - don't recursively traverse every dir
`filepath.Walk` recursively traverses every dir, which is not what is
needed for getPids.
Instead only read the list of dirs in the top level of `/proc`.

```
benchmark              old ns/op     new ns/op     delta
BenchmarkGetPids-4     868684        195522        -77.49%
```

@k8s-ci-robot k8s-ci-robot added size/L and removed size/M labels Jul 19, 2018

@cpuguy83

This comment has been minimized.

Contributor

cpuguy83 commented Jul 19, 2018

Updated, also optimized in case of many entries in /proc so that we don't allocate a huge slice.

@fcrisciani

LGTM

@fcrisciani

This comment has been minimized.

fcrisciani commented Jul 24, 2018

@fejta-bot

This comment has been minimized.

fejta-bot commented Oct 29, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@fcrisciani

This comment has been minimized.

fcrisciani commented Oct 29, 2018

still valid
/remove-lifecycle stale

@dims

This comment has been minimized.

Member

dims commented Oct 29, 2018

/test all
/assign @thockin

@thockin

This comment has been minimized.

Member

thockin commented Oct 29, 2018

Would it be simpler to make a more surgical change? Something like:

    filepath.Walk("/proc", func(path string, info os.FileInfo, err error) error {
        if err != nil {
            // We should continue processing other directories/files
            return nil 
        }
        if path == "/proc" {
            // Skip the top-level dir, but recurse
            return nil 
        }
        fmt.Printf("path %v\n", path)
        if !info.IsDir() {
            // Skip files
            return nil 
        }
        // If the directory is not a number (i.e. not a PID), skip it
        base := filepath.Base(path)
        if _, err := strconv.Atoi(base); err != nil {
            fmt.Printf("skipping %v\n", path)
            return filepath.SkipDir
        }
        fmt.Printf("found pid %v => %v/cmdline\n", base, path)
        // ...process it...
        return filepath.SkipDir // done with this dir, don't recurse further
    })  
@thockin

This comment has been minimized.

Member

thockin commented Oct 29, 2018

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm label Oct 29, 2018

@k8s-ci-robot

This comment has been minimized.

Contributor

k8s-ci-robot commented Oct 29, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cpuguy83, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@thockin

This comment has been minimized.

Member

thockin commented Oct 29, 2018

Did some quick tests, this is better and complete

@k8s-ci-robot k8s-ci-robot merged commit 4c874db into kubernetes:master Oct 29, 2018

11 of 19 checks passed

Submit Queue Milestone is for a future release and cannot be merged
Details
pull-kubernetes-e2e-gce Job triggered.
Details
pull-kubernetes-e2e-gce-100-performance Job triggered.
Details
pull-kubernetes-e2e-kops-aws Job triggered.
Details
pull-kubernetes-integration Job triggered.
Details
pull-kubernetes-kubemark-e2e-gce-big Job triggered.
Details
pull-kubernetes-node-e2e Job triggered.
Details
pull-kubernetes-verify Job triggered.
Details
cla/linuxfoundation cpuguy83 authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Skipped
pull-kubernetes-e2e-kubeadm-gce Skipped
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Skipped
pull-kubernetes-typecheck Job succeeded.
Details
tide In merge pool.
Details

@cpuguy83 cpuguy83 deleted the cpuguy83:speedup_pidof branch Oct 29, 2018

@cpuguy83

This comment has been minimized.

Contributor

cpuguy83 commented Oct 29, 2018

🎉 Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment